Equipment control method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to the technical field of machine learning, and particularly provides an equipment control method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a collected environment image; inputting the environment image into a target equipment control model to obtain a target equipment action; the target equipment control model is obtained based on visual reinforcement learning framework training and comprises a representation extraction module and an action network, the representation extraction module is a pre-trained model which is trained in advance and is used for extracting image representation, and the action network is used for determining a target equipment action according to the image representation; and executing the target equipment action. Therefore, the generalization of the equipment control model can be improved, and the model training cost can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and in particular to a method, apparatus, electronic device, storage medium and program product for controlling a device. Background Art

[0002] In some scenarios, it is usually necessary to use a device control model to control the device to perform corresponding actions according to the state of the external environment. For example, in a robot control scenario, the surrounding environment image is usually collected, and the robot arm's execution action is determined based on the environment image through a trained robot control model (i.e., device control model).

[0003] In related technologies, model training is usually performed for a specific environment to obtain a trained device control model, and the device is controlled to perform corresponding actions based on the environmental image of the specific environment through the device control model.

[0004] However, using this approach, the generalization of the device control model is low, that is, the scope of application is small. It is usually necessary to train the model separately for different application scenarios, and the cost of model training is high. Summary of the invention

[0005] The purpose of the embodiments of the present disclosure is to provide a method, apparatus, electronic device, storage medium and program product for device control, so as to improve the generalization of the device control model and reduce the model training cost.

[0006] In one aspect, an embodiment of the present disclosure provides a method for controlling a device, including:

[0007] Acquire the collected environment image;

[0008] Inputting the environment image into the target device control model to obtain the target device action; the target device control model is obtained by training based on the visual reinforcement learning framework, and includes a representation extraction module and an action network, the representation extraction module is a pre-trained model for extracting image representation, and the action network is used to determine the target device action according to the image representation;

[0009] Execute the target device action.

[0010] In one implementation, inputting the environment image into the target device control model to obtain the target device action includes:

[0011] Inputting the environment image into the representation extraction module to obtain the representation extraction data;

[0012] Inputting the representation extracted data into a multi-layer perceptron also included in the target device control model to obtain perception data;

[0013] The perception data is input into the action network to determine the target device action.

[0014] In one embodiment, the representation extraction module includes a plurality of neural network layers and their respective corresponding batch normalization submodules;

[0015] The multi-layer perceptron includes multiple neural network layers and their corresponding batch normalization submodules;

[0016] The batch normalization submodule is used to perform batch normalization on the output data of the corresponding neural network layer.

[0017] In one implementation, inputting the environment image into the representation extraction module to obtain the representation extraction data includes:

[0018] Input the environment image into the representation extraction module to obtain the image representation output by each of the multiple neural network layers;

[0019] The image representations corresponding to the first n neural network layers in the representation extraction module are used as representation extraction data, where n is a positive integer.

[0020] In one implementation, after executing the target device action, the method further includes:

[0021] Determine the reward corresponding to the target device action;

[0022] Based on the reward, the model parameters in the batch normalization submodule are updated.

[0023] In one embodiment, before inputting the environment image into the target device control model to obtain the target device action, the method further includes: looping through the following steps:

[0024] Input the image sample data into the initial device control model to obtain and execute the corresponding initial device action;

[0025] According to the reward corresponding to the initial device action, it is determined whether the training conditions are met. If so, the initial device control model is used as the target device control model. Otherwise, the model parameters of the initial device control model are updated.

[0026] In one embodiment, before inputting the image sample data into the initial device control model, the method further includes:

[0027] Perform hybrid strong data augmentation and / or weak data augmentation on the image sample data.

[0028] In one embodiment, the initial device control model includes a representation extraction module, a multi-layer perceptron, an action network, and a value evaluation network; the representation extraction module and the multi-layer perceptron both include a plurality of neural network layers and their corresponding batch normalization submodules, and the batch normalization submodule is used to perform batch normalization processing on the output data of the corresponding neural network layer;

[0029] Update the model parameters of the initial equipment control model, including:

[0030] Based on the rewards, model parameters in multiple batch normalization submodules, action networks, and value evaluation networks in the initial device control model are updated.

[0031] In one aspect, an embodiment of the present disclosure provides a device control apparatus, including:

[0032] An acquisition unit, used for acquiring the collected environment image;

[0033] An acquisition unit is used to input the environment image into the target device control model to obtain the target device action; the target device control model is obtained by training based on the visual reinforcement learning framework, including a representation extraction module and an action network, the representation extraction module is a pre-trained model for extracting image representation, and the action network is used to determine the target device action according to the image representation;

[0034] The execution unit is used to execute the target device action.

[0035] In one embodiment, the obtaining unit is used to:

[0036] Inputting the environment image into the representation extraction module to obtain the representation extraction data;

[0037] Inputting the representation extracted data into a multi-layer perceptron also included in the target device control model to obtain perception data;

[0038] The perception data is input into the action network to determine the target device action.

[0039] In one embodiment, the representation extraction module includes a plurality of neural network layers and their respective corresponding batch normalization submodules;

[0040] The multi-layer perceptron includes multiple neural network layers and their corresponding batch normalization submodules;

[0041] The batch normalization submodule is used to perform batch normalization on the output data of the corresponding neural network layer.

[0042] In one embodiment, the obtaining unit is used to:

[0043] Input the environment image into the representation extraction module to obtain the image representation output by each of the multiple neural network layers;

[0044] The image representations corresponding to the first n neural network layers in the representation extraction module are used as representation extraction data, where n is a positive integer.

[0045] In one implementation, the execution unit is further configured to:

[0046] Determine the reward corresponding to the target device action;

[0047] Based on the reward, the model parameters in the batch normalization submodule are updated.

[0048] In one implementation, the obtaining unit is further configured to:

[0049] The following steps are executed in a loop:

[0050] Input the image sample data into the initial device control model to obtain and execute the corresponding initial device action;

[0051] According to the reward corresponding to the initial device action, it is determined whether the training conditions are met. If so, the initial device control model is used as the target device control model. Otherwise, the model parameters of the initial device control model are updated.

[0052] In one implementation, the obtaining unit is further configured to:

[0053] Perform hybrid strong data augmentation and / or weak data augmentation on the image sample data.

[0054] In one embodiment, the initial device control model includes a representation extraction module, a multi-layer perceptron, an action network, and a value evaluation network; the representation extraction module and the multi-layer perceptron both include a plurality of neural network layers and their corresponding batch normalization submodules, and the batch normalization submodule is used to perform batch normalization processing on the output data of the corresponding neural network layer;

[0055] The acquisition unit is also used to update model parameters in multiple batch normalization submodules, action networks, and value evaluation networks in the initial device control model according to the reward.

[0056] In one aspect, an electronic device is provided in an embodiment of the present disclosure, including:

[0057] Processor; and

[0058] A memory stores computer instructions, wherein the computer instructions are used to enable a processor to execute the steps of the method provided in any of the various optional implementations of any of the above-mentioned device controls.

[0059] On the one hand, an embodiment of the present disclosure provides a computer-readable storage medium storing computer instructions, which are used to enable a computer to execute the steps of the method provided in various optional implementations of any of the above-mentioned device control.

[0060] On the one hand, a computer program product is provided in an embodiment of the present disclosure, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the steps of the method provided in various optional implementations of any of the above-mentioned device controls.

[0061] The device control method in the embodiment of the present disclosure includes acquiring a captured environment image; inputting the environment image into a target device control model to obtain a target device action; the target device control model is obtained by training based on a visual reinforcement learning framework, and includes a representation extraction module and an action network, the representation extraction module is a pre-trained model for extracting image representations, and the action network is used to determine the target device action based on the image representation; and the target device action is executed. In this way, the generalization of the device control model can be improved and the model training cost can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a structural diagram of a visual reinforcement learning framework in an embodiment of the present disclosure.

[0063] Figure 2 It is a flowchart of a method for model training in an embodiment of the present disclosure.

[0064] Figure 3 It is an example diagram of characterization comparison in an embodiment of the present disclosure.

[0065] Figure 4 It is a schematic diagram of a characterization extraction in an embodiment of the present disclosure.

[0066] Figure 5 It is a flowchart of a method for controlling a device in an embodiment of the present disclosure.

[0067] Figure 6 It is an application scenario comparison diagram in an embodiment of the present disclosure.

[0068] Figure 7 It is a structural block diagram of a device controlling an apparatus in an embodiment of the present disclosure.

[0069] Figure 8 It is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] The technical solution of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described implementation is a part of the implementation of the present disclosure, rather than all the implementations. Based on the implementation in the present disclosure, all other implementations obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure. In addition, the technical features involved in the different implementations of the present disclosure described below can be combined with each other as long as they do not conflict with each other.

[0071] In the related art, the generalization of the device control model is relatively small, and it is usually necessary to train the model separately for different application scenarios, which results in high model training costs. Based on the defects of the above-mentioned related art, the embodiments of the present disclosure provide a device control method, apparatus, electronic device, storage medium and program product, aiming to improve the generalization of the device control model and reduce the model training costs.

[0072] A method for controlling a device is provided in an embodiment of the present disclosure. The method can be applied to an electronic device. The present disclosure does not limit the type of the electronic device, which can be any type of device suitable for implementation. For example, the electronic device can be a robot, and the type of the robot can be arbitrary, such as robots of different forms. The present disclosure will not elaborate on this.

[0073] In the embodiment of the present disclosure, a model training phase and a model application phase are included. In the model training phase, the initial device control model is trained based on the visual reinforcement learning framework to obtain a trained target device control model. In the model application phase, the trained target device control model is used to execute the corresponding target device action according to the environment image.

[0074] Combine the following Figure 1 For a description of the visual reinforcement learning framework, see Figure 1 As shown, this is a structural diagram of a visual reinforcement learning framework. Figure 1 The device control model includes an image input module, a device control model, an action output module, a value output module, an environment, and a replay buffer. The device control model includes a representation extraction module, a multi-layer perceptron, an action network, and a value evaluation network.

[0075] Among them, the device control model can be an initial device control model or a target device control model. The initial device control model is a model in the training process, and the target device control model is a trained model. The Replay Buffer stores image sample data collected from the environment. The image input module is used to input the image sample data in the Replay Buffer into the initial device control model. The action output module is used to obtain and execute the initial device action output by the initial device control model in the environment, and add it to the Replay Buffer. The value output module is used to obtain the evaluation value output by the initial device control model. Both the representation extraction module and the multilayer perceptron include multiple neural network layers and their corresponding batch normalization submodules, and the batch normalization submodule is used to batch normalize the output data of the corresponding neural network layer.

[0076] The representation extraction module is a pre-trained model for extracting image representations. Pre-trained models are deep learning models that are pre-trained on large-scale datasets. They provide powerful initial parameters for specific tasks by learning common features in large amounts of data. Pre-trained models usually have good generalization capabilities and can be applied to other tasks, thereby reducing training time and computing resource consumption.

[0077] For example, the large-scale dataset may be an image dataset (ImageNet), which is a large-scale dataset for image classification, positioning, and detection.

[0078] Combine the following Figure 1 and Figure 2 , which explains the model training method. Figure 2 As shown, it is a flow chart of a model training method, and the specific implementation process of the method is as follows:

[0079] Step 201: Obtain image sample data.

[0080] In one embodiment, an image acquisition device (such as a camera of a robot) is used to Figure 1 In the Environment, image acquisition is performed to obtain image sample data. The image sample data includes multiple image samples.

[0081] Furthermore, in order to improve generalization, mixed strong data augmentation and / or weak data augmentation may be performed on the image sample data.

[0082] Optionally, the mixed strong data augmentation may be image interpolation mixing of the image sample data, and the weak data augmentation may be image translation of the image sample data.

[0083] Step 202: Input the image sample data into the initial device control model to obtain and execute the corresponding initial device action.

[0084] In one implementation, the image sample data is input into a representation extraction module to obtain the representation extraction data; the representation extraction data is input into a multi-layer perceptron to obtain the perception data; the perception data is input into an action network to obtain an initial device action, and the initial device action is executed in the environment. Furthermore, the perception data is also input into a value evaluation network to obtain an evaluation value.

[0085] See also Figure 3 The figure shows an example of a characterization comparison. Figure 3 In the example, the original image (Original observation) is obtained, and the background in the original image is replaced to obtain a contrast image (Video observation) after the background is replaced. The left side indicates that, through the relevant technology, image representation extraction is performed on the original image and the contrast image respectively, and the obtained original image representation and the first difference map (Feature map difference) of the contrast image are determined. Similarly, the right side indicates that, through the embodiment of the present disclosure, image representation extraction is performed on the original image and the contrast image respectively, and the obtained original image representation and the second difference map of the contrast image are determined. The first difference map is relatively bright, reflecting that under different visual conditions, the feature changes are large and the generalization is weak, while the second difference map is relatively dark, indicating that under different visual conditions, the feature changes are small and the generalization is strong.

[0086] In one implementation, when inputting image sample data into a representation extraction module and obtaining the representation extraction data, the following steps may be used:

[0087] The environment image is input into the representation extraction module to obtain the image representations output by each of the multiple neural network layers; the image representations corresponding to the first n neural network layers in the representation extraction module are used as the representation extraction data, where n is a positive integer. For example, n can be 2.

[0088] See also Figure 4As shown, it is a schematic diagram of a representation extraction. The robot image (i.e., image sample) is input into the representation extraction module including four neural network layers to obtain the image representations output by the first to fourth neural network layers. It can be seen that the image representations corresponding to the first to second neural network layers better retain the robot arm in the robot image and the state information of the robot arm (i.e., position information). This information is of high value for the subsequent determination of the device action, while the image representations corresponding to the third to fourth neural network layers are difficult to identify the robot arm in the robot image, as well as the state information of the robot arm, and mainly contain high-level semantic information, which is of little significance for the determination of the device action. Therefore, in the embodiment of the present disclosure, only the image representations corresponding to the first n neural network layers are selected as the representation extraction data.

[0089] Under the relevant technology, a pre-trained model is usually not used for representation extraction. Instead, a non-pre-trained model is directly used as a visual encoder for feature extraction, and the image representation of the most suitable neural network layer (i.e., the first n neural network layers) is not selected as the representation extraction data. The data processing volume is large, the resource and time costs are high, and it is easy to cause interference with the determination of device actions, and the generalization is low.

[0090] Step 203: Determine the reward corresponding to the initial device action.

[0091] Among them, the reward is the feedback from the environment after performing the initial device action.

[0092] Step 204: Based on the reward, determine whether the training conditions are met. If so, execute step 205; otherwise, execute step 206.

[0093] Step 205: Use the initial device control model as the target device control model.

[0094] Step 206: Update the model parameters of the initial device control model.

[0095] In one implementation, model parameters of the initial device control model are updated based on the reward. Specifically, model parameters of multiple batch normalization submodules, action networks, and value evaluation networks in the initial device control model may be updated based on the reward.

[0096] Furthermore, the model parameters of the initial device control model may be updated in combination with the reward and the evaluation value.

[0097] In this way, model parameters can be updated only for the batch normalization submodule, action network, and value evaluation network.

[0098] See also Figure 5 FIG. 1 is a flow chart of a method for controlling a device in an embodiment of the present disclosure. Figure 5 The method is described below. The specific implementation process of the method is as follows:

[0099] Step 501: Acquire the collected environment image.

[0100] Step 502: Input the environment image into the target device control model to obtain the target device action; the target device control model is obtained by training based on the visual reinforcement learning framework, including a representation extraction module and an action network, the representation extraction module is a pre-trained model for extracting image representation, and the action network is used to determine the target device action based on the image representation.

[0101] In one embodiment, the target device control model also includes a multi-layer perceptron, which inputs the environmental image into the representation extraction module to obtain the representation extraction data; inputs the representation extraction data into the multi-layer perceptron to obtain the perception data; and inputs the perception data into the action network to determine the target device action.

[0102] Among them, the representation extraction module includes multiple neural network layers and their corresponding batch normalization sub-modules; the multilayer perceptron includes multiple neural network layers and their corresponding batch normalization sub-modules; the batch normalization sub-module is used to batch normalize the output data of the corresponding neural network layer.

[0103] In one implementation, when inputting the environment image into the representation extraction module and obtaining the representation extraction data, the following steps may be adopted:

[0104] The environmental image is input into the representation extraction module to obtain the image representations output by each of the multiple neural network layers; the image representations corresponding to the first n neural network layers in the representation extraction module are used as representation extraction data, where n is a positive integer.

[0105] Step 503: Execute the target device action.

[0106] Furthermore, the model parameters in the batch normalization submodule can also be updated.

[0107] In one implementation, a reward corresponding to the target device action is determined; and based on the reward, model parameters in a batch normalization submodule are updated.

[0108] In this way, the model parameters in the batch normalization submodule can be updated during model training and model application.

[0109] See also Figure 6 The figure shows a comparison diagram of application scenarios. Figure 6 There are four scene pictures in the picture. The foreground of each scene picture is the same, which is a robotic arm and a wooden box. The background of each scene picture is different. Figure 6(a) is an example of a standard scene for model training, with a grid street stall as the background. Figure 6 (b)-Scenario Figure 6 (d) is a non-standard visual scene used for model testing. Figure 6 (b)-Scenario Figure 6 The backgrounds in (d) are Metal, Wood, and Blanket. Figure 6 In the scenario shown in (a), the initial device action model is trained to obtain the trained target device action. Then, the trained target device action can be used in the scenario Figure 6 (b)-Scenario Figure 6 The models were tested in the scenario shown in (d), and both had good model performance.

[0110] In the disclosed embodiment, an initial device control model is constructed based on a pre-trained model, and model training is performed based on a visual reinforcement learning framework. The outputs of the first n neural network layers in the pre-trained model are selected as data for subsequent processing. This can improve the generalization of the model and make it applicable to a wider range of untrained scenarios, thereby reducing marginal costs. The model can be deployed in visual environments that have never been seen before and still guarantee good performance. Batch normalization is performed during model training and application, which improves the training efficiency and stability of the model.

[0111] Based on the same inventive concept, a device for device control is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the above-mentioned device and the device is similar to that of a method for device control, the implementation of the above-mentioned device can refer to the implementation of the method, and the repeated parts will not be repeated. The device can be applied to electronic devices. The present disclosure does not limit the type of electronic devices. It can be any type of device suitable for implementation, such as terminal devices and servers, etc. The present disclosure will not repeat them. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the electronic device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution.

[0112] See also Figure 7 FIG. 1 is a block diagram of a device control apparatus according to an embodiment of the present disclosure. In some implementations, the device control apparatus according to an example of the present disclosure includes:

[0113] An acquisition unit 701 is used to acquire a collected environment image;

[0114] The obtaining unit 702 is used to input the environment image into the target device control model to obtain the target device action; the target device control model is obtained by training based on the visual reinforcement learning framework, and includes a representation extraction module and an action network, the representation extraction module is a pre-trained model for extracting image representation, and the action network is used to determine the target device action according to the image representation;

[0115] The execution unit 703 is used to execute the target device action.

[0116] In one implementation, the obtaining unit 702 is used to:

[0117] Inputting the environment image into the representation extraction module to obtain the representation extraction data;

[0118] Inputting the representation extracted data into a multi-layer perceptron also included in the target device control model to obtain perception data;

[0119] The perception data is input into the action network to determine the target device action.

[0120] In one embodiment, the representation extraction module includes a plurality of neural network layers and their respective corresponding batch normalization submodules;

[0121] The multi-layer perceptron includes multiple neural network layers and their corresponding batch normalization submodules;

[0122] The batch normalization submodule is used to perform batch normalization on the output data of the corresponding neural network layer.

[0123] In one implementation, the obtaining unit 702 is used to:

[0124] Input the environment image into the representation extraction module to obtain the image representation output by each of the multiple neural network layers;

[0125] The image representations corresponding to the first n neural network layers in the representation extraction module are used as representation extraction data, where n is a positive integer.

[0126] In one implementation, the execution unit 703 is further configured to:

[0127] Determine the reward corresponding to the target device action;

[0128] Based on the reward, the model parameters in the batch normalization submodule are updated.

[0129] In one implementation, the obtaining unit 702 is further configured to:

[0130] The following steps are executed in a loop:

[0131] Input the image sample data into the initial device control model to obtain and execute the corresponding initial device action;

[0132] According to the reward corresponding to the initial device action, it is determined whether the training conditions are met. If so, the initial device control model is used as the target device control model. Otherwise, the model parameters of the initial device control model are updated.

[0133] In one implementation, the obtaining unit 702 is further configured to:

[0134] Perform hybrid strong data augmentation and / or weak data augmentation on the image sample data.

[0135] In one embodiment, the initial device control model includes a representation extraction module, a multi-layer perceptron, an action network, and a value evaluation network; the representation extraction module and the multi-layer perceptron both include a plurality of neural network layers and their corresponding batch normalization submodules, and the batch normalization submodule is used to perform batch normalization processing on the output data of the corresponding neural network layer;

[0136] The obtaining unit 702 is also used to update the model parameters in the multiple batch normalization submodules, the action network and the value evaluation network in the initial device control model according to the reward.

[0137] The device control method in the embodiment of the present disclosure includes acquiring a captured environment image; inputting the environment image into a target device control model to obtain a target device action; the target device control model is obtained by training based on a visual reinforcement learning framework, and includes a representation extraction module and an action network, the representation extraction module is a pre-trained model for extracting image representations, and the action network is used to determine the target device action based on the image representation; and the target device action is executed. In this way, the generalization of the device control model can be improved and the model training cost can be reduced.

[0138] In an embodiment of the present disclosure, an electronic device is further provided, including:

[0139] Processor; and

[0140] The memory stores computer instructions, where the computer instructions are used to enable the processor to execute the method of any of the above embodiments.

[0141] In an embodiment of the present disclosure, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a computer to execute a method in any of the above embodiments.

[0142] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device implements any of the above-mentioned embodiments of the method.

[0143] Figure 8 FIG. 8 is a schematic diagram showing the structure of an electronic device 8000. Figure 8 As shown, the electronic device 8000 includes: a processor 8010 and a memory 8020 , and optionally, may also include a power supply 8030 , a display unit 8040 , and an input unit 8050 .

[0144] The processor 8010 is the control center of the electronic device 8000. It connects various components using various interfaces and lines, and performs various functions of the electronic device 8000 by running or executing software programs and / or data stored in the memory 8020, thereby monitoring the electronic device 8000 as a whole.

[0145] In the embodiment of the present disclosure, the processor 8010 executes the various steps in the above embodiment when calling the computer program stored in the memory 8020.

[0146] Optionally, the processor 8010 may include one or more processing units; preferably, the processor 8010 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 8010. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips.

[0147] The memory 8020 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, various applications, etc.; the data storage area may store data created according to the use of the electronic device 8000, etc. In addition, the memory 8020 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices, etc.

[0148] The electronic device 8000 also includes a power source 8030 (such as a battery) for supplying power to various components. The power source can be logically connected to the processor 8010 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0149] The display unit 8040 may be used to display information input by the user or information provided to the user and various menus of the electronic device 8000, and in the embodiment of the present disclosure, is mainly used to display the display interface of each application in the electronic device 8000 and objects such as text and pictures displayed in the display interface. The display unit 8040 may include a display panel 8041. The display panel 8041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0150] The input unit 8050 may be used to receive information such as numbers or characters input by the user. The input unit 8050 may include a touch panel 8051 and other input devices 8052. The touch panel 8051, also known as a touch screen, may collect user touch operations on or near it (such as operations performed by the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 8051).

[0151] Specifically, the touch panel 8051 can detect the user's touch operation, detect the signal brought by the touch operation, convert these signals into touch point coordinates, send them to the processor 8010, and receive and execute the command sent by the processor 8010. In addition, the touch panel 8051 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 8052 can include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, power on / off keys, etc.), a trackball, a mouse, a joystick, etc.

[0152] Of course, the touch panel 8051 can cover the display panel 8041. When the touch panel 8051 detects a touch operation on or near it, it transmits it to the processor 8010 to determine the type of touch event, and then the processor 8010 provides corresponding visual output on the display panel 8041 according to the type of touch event. Figure 8 In the embodiment, the touch panel 8051 and the display panel 8041 are used as two independent components to implement the input and output functions of the electronic device 8000, but in some embodiments, the touch panel 8051 and the display panel 8041 can be integrated to implement the input and output functions of the electronic device 8000.

[0153] The electronic device 8000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 8000 may also include other components such as a camera. Since these components are not the key components used in the embodiments of the present disclosure, Figure 8It is not shown and will not be described in detail.

[0154] Those skilled in the art will understand that Figure 8 The electronic device is merely an example and does not limit the electronic device, and may include more or less components than those shown in the figure, or may combine certain components, or may include different components.

[0155] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing the present disclosure, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

Claims

1. A method for controlling a device, characterized in that: The method comprises: Acquire the collected environment image; The environment image is input into a target device control model to obtain a target device action; the target device control model is obtained by training based on a visual reinforcement learning framework, and includes a representation extraction module and an action network, the representation extraction module is a pre-trained model for extracting image representations, and the action network is used to determine the target device action according to the image representation; The target device action is executed.

2. The method according to claim 1, characterized in that The step of inputting the environment image into a target device control model to obtain a target device action includes: Inputting the environment image into the representation extraction module to obtain representation extraction data; Inputting the representation extracted data into a multi-layer perceptron also included in the target device control model to obtain perceptual data; The perception data is input into the action network to determine the target device action.

3. The method according to claim 2, characterized in that The representation extraction module includes a plurality of neural network layers and their corresponding batch normalization submodules; The multi-layer perceptron includes a plurality of neural network layers and respective batch normalization submodules corresponding thereto; The batch normalization submodule is used to perform batch normalization processing on the output data of the corresponding neural network layer.

4. The method according to claim 3, characterized in that The step of inputting the environment image into the representation extraction module to obtain the representation extraction data includes: Inputting the environment image into the representation extraction module to obtain image representations output by each of the multiple neural network layers; The image representations corresponding to the first n neural network layers in the representation extraction module are used as the representation extraction data, where n is a positive integer.

5. The method according to claim 3 or 4, characterized in that: After executing the target device action, the method further includes: Determining a reward corresponding to the target device action; According to the reward, model parameters in the batch normalization submodule are updated.

6. The method according to any one of claims 1 to 4, characterized in that: Before inputting the environment image into the target device control model to obtain the target device action, the method further includes: looping through the following steps: Input the image sample data into the initial device control model to obtain and execute the corresponding initial device action; According to the reward corresponding to the initial device action, it is determined whether the training conditions are met. If so, the initial device control model is used as the target device control model. Otherwise, the model parameters of the initial device control model are updated.

7. The method according to claim 6, characterized in that Before inputting the image sample data into the initial device control model, the method further includes: The image sample data is subjected to mixed strong data augmentation and / or weak data augmentation.

8. The method according to claim 6, characterized in that The initial device control model includes a characterization extraction module, a multi-layer perceptron, an action network, and a value evaluation network; the characterization extraction module and the multi-layer perceptron both include a plurality of neural network layers and their corresponding batch normalization submodules, and the batch normalization submodule is used to perform batch normalization processing on the output data of the corresponding neural network layer; The updating of the model parameters of the initial device control model comprises: According to the reward, model parameters in multiple batch normalization submodules in the initial device control model, the action network, and the value evaluation network are updated.

9. A device for controlling equipment, characterized in that: The device comprises: An acquisition unit, used for acquiring the collected environment image; an obtaining unit, configured to input the environment image into a target device control model to obtain a target device action; the target device control model is obtained by training based on a visual reinforcement learning framework, and includes a representation extraction module and an action network, the representation extraction module is a pre-trained model for extracting image representations, and the action network is used to determine the target device action according to the image representation; An execution unit is used to execute the target device action.

10. An electronic device, characterized in that: include: processor; as well as A memory storing computer instructions, wherein the computer instructions are used to enable the processor to execute the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The invention comprises a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device performs the method according to any one of claims 1 to 8.

Images