Camera control methods

Abstract

The disclosure provides a camera control method, which is executed by a first electronic device. The first electronic device comprises at least a first mechanical arm and a camera connected to the first mechanical arm. The method comprises: obtaining a sample data set of an initial neural network model, wherein the initial neural network model is used to generate a control signal for driving the first mechanical arm; training the initial neural network model according to the sample data set to obtain a target neural network model; and generating a target control signal according to the target neural network model, wherein the target control signal is used to drive the first mechanical arm to control the camera to obtain target image data of an object to be identified. Thus, the target control signal for the first mechanical arm can be quickly and accurately generated based on the target neural network model, the camera can obtain the target image data based on an optimal viewing angle, and the identification effect of the object to be identified based on the target image data can be ensured.

Description

Technical Field

[0001] This disclosure relates to the field of electronic device technology, and specifically to a method for controlling a camera. Background Technology

[0002] Currently, mobile terminal-based inspection robot systems are gradually being adopted in industries such as chemical engineering, power grids, and coal mining. These platforms use wheeled or legged mobile platforms and are equipped with pan-tilt units carrying various monitoring instruments, among which vision-based instrument monitoring is more commonly used. In some scenarios, it is necessary to point vision instruments (such as cameras) at the instruments to obtain higher reading accuracy.

[0003] In related technologies, when a camera is aimed at an instrument, the image data acquired by the camera may be poor, which may affect the accuracy of the readings of the instrument. Summary of the Invention

[0004] This disclosure aims to at least partially address one of the technical problems in the related art.

[0005] Therefore, the purpose of this disclosure is to propose a camera control method, device, electronic device and storage medium that can quickly and accurately generate target control signals for a first robotic arm based on a target neural network model, so as to enable the camera to acquire target image data from a better perspective, thereby ensuring the recognition effect of the object to be recognized based on the target image data.

[0006] The camera control method proposed in the first aspect of this disclosure is executed by a first electronic device, the first electronic device including at least: a first robotic arm, and a camera connected to the first robotic arm, the method including:

[0007] Obtain a sample data set of an initial neural network model, wherein the initial neural network model is used to generate control signals to drive the first robotic arm;

[0008] The initial neural network model is trained based on the sample data set to obtain the target neural network model;

[0009] Based on the target neural network model, a target control signal is generated, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified.

[0010] The camera control method proposed in the first aspect of this disclosure acquires a sample data set of an initial neural network model, wherein the initial neural network model is used to generate control signals to drive a first robotic arm. The initial neural network model is trained based on the sample data set to obtain a target neural network model. Based on the target neural network model, a target control signal is generated, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified. Thus, a target control signal for the first robotic arm can be generated quickly and accurately based on the target neural network model, enabling the camera to acquire target image data from a better perspective, thereby ensuring the recognition effect of the object to be identified based on the target image data.

[0011] The camera control device according to the second aspect embodiment of this disclosure is executed by a first electronic device, the first electronic device including at least: a first robotic arm, and a camera connected to the first robotic arm, the device including:

[0012] An acquisition module is used to acquire a sample data set of an initial neural network model, wherein the initial neural network model is used to generate control signals to drive the first robotic arm;

[0013] The model training module is used to train the initial neural network model based on the sample data set to obtain the target neural network model;

[0014] The generation module is used to generate a target control signal based on the target neural network model, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified.

[0015] The camera control device proposed in the second aspect of this disclosure acquires a sample data set of an initial neural network model, wherein the initial neural network model is used to generate control signals to drive a first robotic arm. The initial neural network model is trained based on the sample data set to obtain a target neural network model. Based on the target neural network model, a target control signal is generated, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified. Thus, a target control signal for the first robotic arm can be generated quickly and accurately based on the target neural network model, enabling the camera to acquire target image data from a better perspective, thereby ensuring the recognition effect of the object to be identified based on the target image data.

[0016] The electronic device proposed in the third aspect of this disclosure includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the camera control method proposed in the first aspect of this disclosure.

[0017] The fourth aspect of this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements a camera control method as described in the first aspect of this disclosure.

[0018] A fifth aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs a camera control method as described in a first aspect of this disclosure.

[0019] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0021] Figure 1 This is a schematic flowchart of a camera control method according to an embodiment of the present disclosure;

[0022] Figure 2 This is a schematic flowchart of a camera control method according to another embodiment of this disclosure;

[0023] Figure 3 This is a schematic flowchart of a camera control method according to another embodiment of this disclosure;

[0024] Figure 4 This is a schematic diagram of the structure of a data collection platform proposed in an embodiment of this disclosure;

[0025] Figure 5 This is a schematic diagram of a deep reinforcement learning training framework proposed in an embodiment of this disclosure;

[0026] Figure 6 This is a schematic diagram of the structure of a camera control device according to an embodiment of the present disclosure;

[0027] Figure 7 This is a schematic diagram of the structure of a camera control device according to another embodiment of this disclosure;

[0028] Figure 8 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0029] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are used only to explain this disclosure, and should not be construed as limiting this disclosure. Rather, embodiments of this disclosure include all variations, modifications, and equivalents falling within the spirit and scope of the appended claims.

[0030] Figure 1 This is a schematic flowchart of a camera control method according to an embodiment of the present disclosure.

[0031] It should be noted that the execution subject of the camera control method in this embodiment is the camera control device, which can be implemented by software and / or hardware. The device can be configured in an electronic device, which includes at least: a first robotic arm and a camera connected to the first robotic arm.

[0032] like Figure 1 As shown, the control method for this camera includes:

[0033] S101: Obtain the sample data set of the initial neural network model, wherein the initial neural network model is used to generate control signals to drive the first robotic arm.

[0034] Here, a neural network model refers to a model built based on a neural network, which can be used as a neural network controller for the first robotic arm to generate control signals to drive the first robotic arm. An initial neural network model, on the other hand, refers to an untrained neural network model.

[0035] Here, sample data refers to the data used to train the initial neural network model mentioned above. A sample data set refers to a collection consisting of at least one sample data set; for example, each sample data set may contain 1 to 10 sample data sets.

[0036] The first robotic arm refers to the robotic arm whose end is connected to the camera, and this robotic arm can be composed of any number of joints.

[0037] In other words, in this embodiment of the present disclosure, the camera can be configured at the end of the first robotic arm, and then an initial neural network model is used to generate control signals to drive the first robotic arm, thereby realizing the spatial movement of the camera to adjust the camera's visual angle.

[0038] Optionally, in some embodiments, when obtaining the sample data set of the initial neural network model, the control period of the initial neural network model can be determined, and the sample data of the initial neural network model can be obtained according to the control period. In response to the sample data satisfying the first preset condition, a sample data set can be constructed based on at least one sample data. Thus, the standardization of the obtained sample data can be effectively improved based on the control period, and the sample data set can be constructed in a timely manner through the first preset condition, which can effectively improve the practicality of the obtained sample data set.

[0039] The control cycle refers to the execution cycle of the control signals generated by the initial neural network model, which is the time it takes for the first robotic arm to execute the aforementioned control signals.

[0040] The first preset condition refers to the threshold condition configured for the sample data collection process during the construction of the sample data set, which can be used to determine whether to stop collecting sample data.

[0041] It is understandable that during the sample data collection process, the collected sample data may meet the requirements of model training in advance. Therefore, a reliable basis for stopping the collection of sample data can be provided based on the first preset condition, which can avoid collecting redundant sample data.

[0042] S102: Train an initial neural network model based on the sample data set to obtain the target neural network model.

[0043] The target neural network model refers to the neural network model obtained by training the initial neural network model based on a set of sample data.

[0044] In this embodiment of the disclosure, when training an initial neural network model based on a sample data set to obtain a target neural network model, the model parameters of the initial neural network model can be adjusted based on a deep reinforcement learning algorithm to obtain a neural network model that can meet the camera control requirements as the target neural network model.

[0045] S103: Generate a target control signal based on the target neural network model, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified.

[0046] Among them, the target control signal refers to the control signal generated based on the target neural network model.

[0047] The object to be identified refers to the object in the image data acquired by the camera that needs to be identified, such as instruments and meters.

[0048] Among them, target image data refers to the image data of the object to be identified acquired by the camera after the first robotic arm is driven based on the target control signal.

[0049] In other words, in this embodiment of the present disclosure, after training an initial neural network model based on a sample data set to obtain a target neural network model, a target control signal can be generated based on the target neural network model in a practical application scenario, thereby effectively improving the automation level and control effect of the camera control process.

[0050] In this embodiment, a sample data set of an initial neural network model is obtained, wherein the initial neural network model is used to generate control signals to drive the first robotic arm. The initial neural network model is trained based on the sample data set to obtain a target neural network model. Based on the target neural network model, a target control signal is generated, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified. Thus, a target control signal for the first robotic arm can be generated quickly and accurately based on the target neural network model, enabling the camera to acquire target image data from a better perspective, thereby ensuring the recognition effect of the object to be identified based on the target image data.

[0051] Figure 2 This is a schematic flowchart of a camera control method according to another embodiment of this disclosure.

[0052] like Figure 2 As shown, the control method for this camera includes:

[0053] S201: Determine the control period of the initial neural network model.

[0054] For a detailed description of S201, please refer to the above embodiments, which will not be repeated here.

[0055] S202: According to the control cycle, obtain the initial state value of the first electronic device in the initial state.

[0056] The initial state refers to the state of the first electronic device before it executes the control signal.

[0057] The initial state value refers to the value generated based on the relevant information of the initial state.

[0058] In this embodiment of the disclosure, when obtaining the initial state value of the first electronic device in the initial state according to the control cycle, a point cloud data acquisition device can be used to obtain point cloud data in the scene where the first electronic device is located, and the initial state value can be determined based on the obtained point cloud data. Alternatively, the state type of the first electronic device can be determined, and then the state value corresponding to the above state type can be determined as the initial state value based on a preset relationship table. There are no restrictions on this.

[0059] Optionally, in some embodiments, the first robotic arm includes at least one joint. When acquiring the initial state value of the first electronic device in the initial state according to the control cycle, the initial angle data of at least one joint can be acquired in the initial state, a first vector can be generated based on the initial angle data, initial image data of the object to be identified can be acquired based on the camera in the initial state, a second vector can be generated based on the initial image data, and the initial state value can be determined based on the first vector and the second vector. Thus, the relevant data of the first robotic arm and the camera can be effectively combined in the process of determining the initial state value, which can effectively improve the state representation effect of the obtained initial state value on the first robotic arm and the camera.

[0060] Here, the initial angle data refers to the angle values ​​of the joints of the first robotic arm in their initial state. The first vector, on the other hand, refers to the vector generated based on the angle values ​​of all joints of the first robotic arm.

[0061] Initial image data refers to the image data acquired by the camera in its initial state.

[0062] In this embodiment of the disclosure, when generating the second vector based on the initial image data, the brightness features of the initial image data may be determined, and the second vector may be generated based on the obtained brightness features.

[0063] S203: Generate sample control signals corresponding to the initial state values ​​based on the initial neural network.

[0064] Among them, the sample control signal refers to the control signal generated by the initial neural network in response to the initial state value.

[0065] Optionally, in some embodiments, when generating sample control signals corresponding to initial state values ​​based on the initial neural network, the spatial position information and spatial posture information of the object to be identified can be determined. In response to the sample posture information satisfying a fourth preset condition, the spatial position information and spatial posture information are processed based on the initial neural network to generate sample control signals. In response to the sample posture information not satisfying the fourth preset condition, the reachable space range and joint adjustment range of the second robotic arm are determined, and the spatial position information and spatial posture information of the object to be identified are adjusted according to the reachable space range and joint adjustment range. The second robotic arm is connected to the object to be identified, and the second robotic arm and the object to be identified together constitute a second electronic device. Thus, based on the fourth preset condition, it can be ensured that the identification information of the object to be identified can be obtained after the sample control signal is executed, which can effectively improve the reliability of the sample control signal generation process.

[0066] Spatial location information can refer to the spatial coordinates of the object to be identified.

[0067] Spatial pose information refers to the pose information of the object to be identified.

[0068] For example, in this embodiment of the disclosure, a spatial coordinate system can be established, and the spatial coordinate data of the object to be identified can be obtained as the above-mentioned spatial position information; the deflection angles (α, β, γ) of the object to be identified in the x-axis, y-axis, and z-axis directions can be obtained as the above-mentioned spatial attitude information.

[0069] The fourth preset condition refers to the condition configured for the sample pose information of the object to be identified before the sample control signal. For example, it can be a range of values ​​configured for the sample pose information, such as α∈(-π,π), β∈(-π / 4,π / 4), γ∈(-π / 4,π / 4).

[0070] The reachable space refers to the spatial range in which the second robotic arm can move.

[0071] S204: Determine the joint adjustment data of the first robotic arm based on the sample control signal.

[0072] Among them, joint adjustment data may refer, for example, to the angle change data of each joint of the first robotic arm during the execution of the sample control signal.

[0073] S205: Drive the first robotic arm based on joint adjustment data to put the first electronic device in a sample state.

[0074] The sample state refers to the state of the first electronic device after the first robotic arm executes the sample control signal.

[0075] Optionally, in some embodiments, when driving the first robotic arm based on joint adjustment data to put the first electronic device in a sample state, the trajectory information of at least one joint during the joint adjustment process can be determined based on the joint adjustment data, point cloud data of the scene where the first electronic device is located can be obtained, and a collision detection result corresponding to the joint adjustment data can be generated based on the trajectory information and the point cloud data. In response to the collision detection result satisfying a second preset condition, the first robotic arm can be driven based on the joint adjustment data to put the first electronic device in a sample state. Thus, a reliable triggering basis can be provided for the execution process of the sample control signal based on the collision detection result, which can effectively improve the safety of the first electronic device in the process of executing the sample control signal.

[0076] The trajectory information refers to the relevant information about the running trajectory of each joint of the first robotic arm during the execution of the sample control signal.

[0077] Point cloud data refers to data generated based on the point cloud of the scene in which the first electronic device is located. It can be used for obstacle identification and localization.

[0078] The collision detection result refers to the prediction of whether a collision will occur during the execution of the sample control signal by the first electronic device, based on trajectory information and point cloud data.

[0079] S206: Determine the sample state value and reward value corresponding to the joint adjustment data, wherein the initial state value, joint adjustment data, sample state value and reward value are combined as sample data.

[0080] The sample state value refers to the state value of the first electronic device in the sample state.

[0081] The reward value is the value of the reward function obtained based on joint adjustment data during model training. The reward function is a function constructed during model training to ensure that the model training results closely resemble the user's intent.

[0082] In other words, after determining the control cycle of the initial neural network model, this embodiment of the present disclosure can obtain the initial state value of the first electronic device in the initial state according to the control cycle, generate a sample control signal corresponding to the initial state value based on the initial neural network, determine the joint adjustment data of the first robotic arm according to the sample control signal, drive the first robotic arm based on the joint adjustment data to make the first electronic device in the sample state, and determine the sample state value and reward value corresponding to the joint adjustment data. The initial state value, joint adjustment data, sample state value and reward value are combined as sample data. Thus, multiple dimensions of relevant data can be effectively combined in the process of generating sample data, which can effectively improve the training effect of the obtained sample data on the initial neural network model.

[0083] S207: In response to the sample data satisfying a first preset condition, construct a sample data set based on at least one sample data.

[0084] Optionally, in some embodiments, the first preset condition may be that the number of sample data is greater than or equal to a first preset value and / or the reward value of the sample data is greater than or equal to a second preset value.

[0085] The first preset value refers to the threshold value of the sample data configured for each sample dataset, such as 10.

[0086] The second preset value refers to the threshold value pre-configured for the reward value of the sample data, such as 5.

[0087] S208: Train an initial neural network model based on the sample data set to obtain the target neural network model.

[0088] S209: Generate a target control signal based on the target neural network model, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified.

[0089] The descriptions of S208-S209 can be found in the above embodiments, and will not be repeated here.

[0090] In this embodiment, by obtaining the initial state value of the first electronic device in the initial state according to the control cycle, generating a sample control signal corresponding to the initial state value based on the initial neural network, determining the joint adjustment data of the first robotic arm based on the sample control signal, driving the first robotic arm based on the joint adjustment data to make the first electronic device in the sample state, and determining the sample state value and reward value corresponding to the joint adjustment data, wherein the initial state value, joint adjustment data, sample state value, and reward value are jointly used as sample data. Therefore, multiple dimensions of relevant data can be effectively combined during the generation of sample data, effectively improving the training effect of the obtained sample data on the initial neural network model. By obtaining the initial angle data of at least one joint in the initial state, generating a first vector based on the initial angle data, acquiring the initial image data of the object to be identified based on the camera in the initial state, generating a second vector based on the initial image data, and determining the initial state value based on the first and second vectors, the relevant data of the first robotic arm and the camera can be effectively combined during the determination of the initial state value, effectively improving the state representation effect of the obtained initial state value on the first robotic arm and the camera. By determining the trajectory information of at least one joint during the joint adjustment process based on the joint adjustment data, point cloud data of the scene where the first electronic device is located is obtained. Based on the trajectory information and point cloud data, a collision detection result corresponding to the joint adjustment data is generated. In response to the collision detection result satisfying the second preset condition, the first robotic arm is driven based on the joint adjustment data to put the first electronic device in the sample state. Thus, a reliable triggering basis can be provided for the execution process of the sample control signal based on the collision detection result, which can effectively improve the safety of the first electronic device in the process of executing the sample control signal.

[0091] Figure 3 This is a schematic flowchart of a camera control method according to another embodiment of this disclosure.

[0092] like Figure 3 As shown, the control method for this camera includes:

[0093] S301: Determine the control period of the initial neural network model.

[0094] S302: According to the control cycle, obtain the initial state value of the first electronic device in the initial state.

[0095] S303: Generate sample control signals corresponding to the initial state values ​​based on the initial neural network.

[0096] S304: Determine the joint adjustment data of the first robotic arm based on the sample control signal.

[0097] S305: Drive the first robotic arm based on joint adjustment data to put the first electronic device in a sample state.

[0098] For details on S301-S305, please refer to the above embodiments, and they will not be repeated here.

[0099] S306: Obtain sample angle data of at least one joint in the sample state.

[0100] Among them, the sample angle data refers to the angle data of each joint of the first robotic arm in the sample state.

[0101] S307: Generate a third vector based on the sample angle data.

[0102] The third vector refers to the vector generated based on the sample angle data of each joint of the first section of the robotic arm.

[0103] S308: In sample state, acquire sample image data of the object to be identified based on the camera.

[0104] Among them, sample image data refers to image data of the object to be identified acquired by the camera in the sample state.

[0105] S309: Generate the fourth vector based on the sample image data.

[0106] The fourth vector refers to a vector generated based on the sample image data. For example, it could be a vector based on the RGB values ​​of the sample image data.

[0107] S310: Determine the sample state value based on the third and fourth vectors.

[0108] S311: Determine the reward value based on the collision detection results.

[0109] Optionally, in some embodiments, when determining the reward value based on the collision detection result, it may be in response to the collision detection result meeting the second preset condition, obtaining the joint adjustment speed value and object recognition error value corresponding to the joint adjustment data, and determining the reward value based on the joint adjustment speed value and object recognition error value; in response to the collision detection result not meeting the second preset condition, using the third preset value as the reward value. Thus, the corresponding reward value can be flexibly determined for different application scenarios, which can effectively improve the adaptability between the obtained reward value and the personalized application scenario, and improve the indicative effect of the reward value on model training.

[0110] The joint adjustment speed value can refer to the rate of change of the angle of each joint of the first robotic arm from the initial angle data to the sample angle data within the control cycle.

[0111] Among them, the object recognition error value refers to the error value between the recognition result of the object to be recognized obtained based on the sample control signal and the actual result.

[0112] For example, the object to be identified may be an instrument. In this embodiment of the disclosure, the reading can be identified by acquiring an image of the instrument through a camera, and then the actual reading of the instrument can be obtained. The object identification error is determined based on the difference between the two readings.

[0113] The second preset condition refers to the restrictive conditions pre-configured based on the collision detection results. For example, it could be that the first robotic arm will not collide during the execution of the sample control signal, or that the minimum distance between the first robotic arm and the obstacle during the execution of the sample control signal is 5cm.

[0114] Optionally, in some embodiments, when obtaining the joint adjustment speed value and object recognition error value corresponding to the joint adjustment data, and determining the reward value based on the joint adjustment speed value and object recognition error value, the joint adjustment speed value can be determined based on the control cycle and the joint adjustment data. The actual recognition value of the object to be recognized can be obtained, and the sample recognition value corresponding to the joint adjustment data can be obtained. Based on the actual recognition value and the sample recognition value, the object recognition error value can be determined. In response to the object recognition error value satisfying the third preset condition, the calculation result of the joint adjustment speed value and the object recognition error value based on the preset calculation template can be used as the reward value. In response to the object recognition error value not satisfying the third preset condition, the fourth preset value can be used as the reward value. Thus, the accuracy of the obtained joint adjustment speed value and object recognition error value can be effectively improved.

[0115] The third preset condition refers to the threshold condition configured in advance for the object recognition error value, such as limiting the threshold range of the object recognition error value.

[0116] The fourth preset value refers to the reward value configured in advance for situations where the object recognition error value does not meet the third preset condition.

[0117] For example, the third preset condition could be that the object recognition error value is greater than the preset value δ. When the object recognition error value is less than or equal to the preset value δ, the fourth preset value is 5 as the reward value.

[0118] In other words, in this embodiment of the present disclosure, after generating a sample control signal corresponding to the initial state value based on spatial position information and spatial attitude information, sample angle data of at least one joint in the sample state can be obtained. Based on the sample angle data, a third vector is generated. In the sample state, sample image data of the object to be identified is obtained based on the camera. Based on the sample image data, a fourth vector is generated. Based on the third and fourth vectors, the sample state value is determined. Based on the collision detection result, the reward value is determined. Thus, the accuracy of the obtained sample state value and reward value can be effectively improved.

[0119] S312: In response to the sample data satisfying a first preset condition, construct a sample data set based on at least one sample data.

[0120] S313: Train an initial neural network model based on the sample data set to obtain the target neural network model.

[0121] S314: Generate a target control signal based on the target neural network model, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified.

[0122] For a detailed description of S312-S314, please refer to the above embodiments, which will not be repeated here.

[0123] In this embodiment, sample angle data of at least one joint in a sample state is acquired. Based on the sample angle data, a third vector is generated. In the sample state, sample image data of the object to be identified is acquired based on a camera. Based on the sample image data, a fourth vector is generated. Based on the third and fourth vectors, a sample state value is determined. Based on the collision detection result, a reward value is determined. This effectively improves the accuracy of the obtained sample state value and reward value. In response to the collision detection result satisfying a second preset condition, the joint adjustment speed value and object recognition error value corresponding to the joint adjustment data are acquired. The reward value is determined based on the joint adjustment speed value and object recognition error value. In response to the collision detection result not satisfying the second preset condition, a third preset value is used as the reward value. Therefore, the corresponding reward value can be flexibly determined for different application scenarios, effectively improving the adaptability of the obtained reward value to personalized application scenarios and enhancing the indicative effect of the reward value on model training. By determining the joint adjustment speed value based on the control cycle and joint adjustment data, the actual recognition value of the object to be identified is obtained, and the sample recognition value corresponding to the joint adjustment data is obtained. Based on the actual recognition value and the sample recognition value, the object recognition error value is determined. If the object recognition error value meets the third preset condition, the calculation result of the joint adjustment speed value and the object recognition error value is obtained based on the preset calculation template as a reward value. If the object recognition error value does not meet the third preset condition, the fourth preset value is used as the reward value. This effectively improves the accuracy of the obtained joint adjustment speed value and object recognition error value. By determining the spatial position information and spatial attitude information of the object to be identified, and generating a sample control signal corresponding to the initial state value based on the spatial position information and spatial attitude information, the robustness of the sample control signal generation process can be effectively improved. By responding to the sample posture information satisfying the fourth preset condition, the initial neural network processes spatial position and spatial posture information to generate a sample control signal. If the sample posture information does not satisfy the fourth preset condition, the reachable space range and joint adjustment range of the second robotic arm are determined. Based on the reachable space range and joint adjustment range, the spatial position and spatial posture information of the object to be identified are adjusted. The second robotic arm is connected to the object to be identified, and the second robotic arm and the object to be identified together constitute the second electronic device. Thus, based on the fourth preset condition, it can be ensured that the sample control signal can obtain the identification information of the object to be identified after execution, which can effectively improve the reliability of the sample control signal generation process.

[0124] For example, in this embodiment of the disclosure, when acquiring the sample data set, it can be based on a pre-configured data collection platform, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a data collection platform proposed in an embodiment of this disclosure, wherein:

[0125] Both robotic arms 1 and 2 are six-degree-of-freedom robotic arms. Robotic arm 1 is equipped with an instrument reading recognition camera at its end, and robotic arm 2 is equipped with a simulated instrument panel to be detected at its end. A depth camera is located behind robotic arm 1, and its field of view covers both robotic arms 1 and 2. This layout simulates the layout of an actual mobile monitoring platform.

[0126] Analog pointer-type instruments need to be equipped with miniature motors, allowing computer-controlled modification of the pointer position, with the precise position of the pointer directly readable by the computer after modification. If simulating a digital instrument, it should also possess the ability to automatically adjust and acquire readings. This function primarily facilitates the collection of data from real instruments and the automatic adjustment for different instrument reading tasks.

[0127] Robotic arm 2 can change the spatial pose of the instrument, simulating different initial relative spatial poses of the instrument and the recognition camera. The initial pose of the instrument is facing the positive x-axis in the diagram, and the z-axis of the coordinate system in the diagram points out of the paper.

[0128] The entire platform workflow is as follows:

[0129] Step 1: Randomly sample the position (x, y, z) and orientation (α, β, γ) of the end effector within the reachable space of robotic arm 2. The position is the relative displacement with respect to the base coordinate system of robotic arm 2, and the three orientation values ​​represent the angles of rotation of the instrument around the x-axis, y-axis, and z-axis in the diagram, respectively.

[0130] Step 2: Check that the instrument attitude is within the readable range, α∈(-π, π), β∈(-π / 4, π / 4), γ∈(-π / 4, π / 4). Within this attitude range, the instrument should be roughly facing the direction of robot arm 1 (i.e. facing the right side of the figure). If it is not within this range, return to step 1.

[0131] Step 3: Use a reinforcement learning neural network controller to control the movement of robotic arm 1, and send a new control signal every 1 second.

[0132] Step 4: Based on the point cloud data of all devices captured by the depth camera, use a collision detection algorithm to detect whether a collision will occur during the execution of control signals by robotic arm 1. If a collision is predicted, end the data collection for this round and return to step 1.

[0133] Step 5: Execute the control signal to start the movement, and after the movement is completed, use the instrument panel recognition software to recognize the instrument panel readings on the current screen.

[0134] Step 6: Repeat steps 3 to 5 until the reading error read by the dashboard recognition software is less than the allowable error δ, or the movement time exceeds 10 seconds.

[0135] Step 7: Repeat steps 1 through 6 to collect data until the deep reinforcement learning training is complete.

[0136] For example, in an embodiment of this disclosure, the target neural network model is an end-to-end controller based on a neural network, and the training framework of the controller is as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of a deep reinforcement learning training framework proposed in an embodiment of this disclosure, wherein:

[0137] The neural network controller is responsible for processing environmental information and outputting control signals. This controller uses a fully connected neural network structure.

[0138] Each training round consists of 10 control cycles. At the start of each control cycle, which has a 1-second interval, the neural network obtains the current state value s from the instrument reading recognition training environment simulation platform. i = (θ, P) as input, i represents the i-th control cycle, θ is a vector composed of the current values ​​of the six joint angles of robotic arm 1, θ = (θ1, θ2, θ3, θ4, θ5, θ6, ), and P is a vector composed of the current RGB values ​​of the camera image.

[0139] The neural network output is a control signal vector Δθ i θ represents the change in the angles of the six joints of the robotic arm, that is, the change in the joint angles of the robotic arm from θ to θ within one second. i Change to θ i+1 =θ i +Δθ i Common robotic arms are equipped with corresponding linear interpolation and control interfaces, which can interpolate to obtain a series of intermediate joint angle values. These intermediate states require collision detection using a collision detection library. If no collision is detected during the movement, the series of actions is executed; otherwise, the training round is terminated. Open-source algorithms, such as the Open Motion Planning Library (OMPL), are typically used for the collision detection portion. At the end of each control cycle, a reward function r is fed back to the reinforcement learning training algorithm. i The definition is as follows:

[0140] When a collision detection indicates that a collision is likely, r i =-5;

[0141] When r a When r > δ, i =1-r a +r v ;

[0142] When r a <δ or r a When r = δ, i =5.

[0143] Where r a r is the absolute value of the relative error of the dashboard recognition algorithm when recognizing instrument readings using the current camera image, and is restricted to a value within the range [0, 1]. v This is a penalty event for the rotational speed of the six joints of the robotic arm within one second during the control process. If any joint of the robotic arm exceeds the specified speed while completing the control signal, then r... v Take -0.5, otherwise take 0. If r a If the error is within the allowable range δ, it is considered that the robotic arm has moved the camera to a suitable position for reading instrument readings, and a reward of 5 is given and the training round is terminated.

[0144] Therefore, the reinforcement learning training data for each control cycle is (s) i a i r i s i+1 Each training run collects no more than 10 sets of training data. i a i r i s i+1 )|i=1,...,10}.

[0145] After collecting enough reinforcement learning training data, the parameters of the neural network controller can be trained and adjusted using reinforcement learning training algorithms. Typical training steps are as follows:

[0146] Step 1: Use parameter p j The neural network controller performs the task of moving the camera to complete the reading in the instrument reading task environment, i.e. the aforementioned process, and collects reinforcement learning training data. In the first round of training, the initial parameters of the neural network are random values.

[0147] Step 2: Using the collected training data, employ reinforcement learning algorithms such as Q-Learning, Soft Actor Criticism, or Proximal Policy Optimization—currently the most commonly used deep reinforcement learning training algorithms—to adjust the parameters of the neural network controller based on the collected training data, thereby obtaining a new neural network controller with parameters p. j+1 ;

[0148] Step 3: Return to Step 1, use the new neural network controller to continue performing the instrument reading recognition task, collect more new reinforcement learning training data, and repeat the above steps until the neural network controller can control the robotic arm 1 to move the camera so that the instrument reading recognition algorithm can stably recognize the instrument reading within the error range.

[0149] Figure 6 This is a schematic diagram of the structure of a camera control device according to an embodiment of the present disclosure.

[0150] like Figure 6 As shown, the camera's control device 60 is executed by a first electronic device, which includes at least: a first robotic arm and a camera connected to the first robotic arm. The device includes:

[0151] The acquisition module 601 is used to acquire a sample data set of the initial neural network model, wherein the initial neural network model is used to generate control signals to drive the first robotic arm;

[0152] The model training module 602 is used to train an initial neural network model based on a set of sample data to obtain a target neural network model.

[0153] The generation module 603 is used to generate a target control signal based on the target neural network model, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified.

[0154] In some embodiments of this disclosure, such as Figure 7 As shown, Figure 7 This is a schematic diagram of the control device for a camera according to another embodiment of the present disclosure. The acquisition module 601 includes:

[0155] Determine submodule 6011, which is used to determine the control cycle of the initial neural network model;

[0156] The acquisition submodule 6012 is used to acquire sample data of the initial neural network model according to the control cycle;

[0157] A submodule 6013 is constructed to construct a sample data set based on at least one sample data in response to the sample data meeting a first preset condition.

[0158] In some embodiments of this disclosure, the acquisition submodule 6012 is specifically used for:

[0159] According to the control cycle, obtain the initial state value of the first electronic device in the initial state;

[0160] The initial neural network generates sample control signals corresponding to the initial state values.

[0161] Based on the sample control signals, determine the joint adjustment data of the first robotic arm;

[0162] The first robotic arm is driven based on joint adjustment data to put the first electronic device in a sample state;

[0163] Determine the sample state value and reward value corresponding to the joint adjustment data, wherein the initial state value, joint adjustment data, sample state value and reward value are combined as sample data.

[0164] In some embodiments of this disclosure, the first preset condition includes any one of the following:

[0165] The number of sample data is greater than or equal to the first preset value;

[0166] The reward value of the sample data is greater than or equal to the second preset value.

[0167] In some embodiments of this disclosure, the first robotic arm includes at least one joint;

[0168] The acquisition submodule 6012 is also used for:

[0169] Acquire initial angle data for at least one joint in the initial state;

[0170] Generate the first vector based on the initial angle data;

[0171] In the initial state, initial image data of the object to be identified is acquired based on the camera;

[0172] Generate a second vector based on the initial image data;

[0173] Determine the initial state value based on the first vector and the second vector.

[0174] In some embodiments of this disclosure, the acquisition submodule 6012 is further used for:

[0175] Based on the joint adjustment data, determine the trajectory information of at least one joint during the joint adjustment process;

[0176] Obtain point cloud data of the scene where the first electronic device is located;

[0177] Based on trajectory information and point cloud data, generate collision detection results corresponding to joint adjustment data;

[0178] In response to the collision detection result meeting the second preset condition, the first robotic arm is driven based on the joint adjustment data to put the first electronic device in the sample state.

[0179] In some embodiments of this disclosure, the acquisition submodule 6012 is further used for:

[0180] Obtain sample angle data for at least one joint in the sample state;

[0181] Generate a third vector based on the sample angle data;

[0182] In the sample state, sample image data of the object to be identified is acquired based on the camera;

[0183] Generate a fourth vector based on the sample image data;

[0184] The sample state value is determined based on the third and fourth vectors;

[0185] The reward value is determined based on the collision detection results.

[0186] In some embodiments of this disclosure, the acquisition submodule 6012 is further used for:

[0187] In response to the collision detection result meeting the second preset condition, the joint adjustment speed value and object recognition error value corresponding to the joint adjustment data are obtained, and the reward value is determined based on the joint adjustment speed value and object recognition error value.

[0188] In response to the collision detection result not meeting the second preset condition, the third preset value is used as the reward value.

[0189] In some embodiments of this disclosure, the acquisition submodule 6012 is further used for:

[0190] Determine the joint adjustment speed value based on the control cycle and joint adjustment data;

[0191] Obtain the actual identification value of the object to be identified;

[0192] Obtain the sample identification value corresponding to the joint adjustment data;

[0193] The object recognition error value is determined based on the actual recognition value and the sample recognition value;

[0194] In response to the object recognition error value meeting the third preset condition, the calculation result of the joint adjustment speed value and the object recognition error value is obtained as a reward value based on the preset calculation template;

[0195] In response to the object recognition error value not meeting the third preset condition, the fourth preset value is used as the reward value.

[0196] In some embodiments of this disclosure, the acquisition submodule 6012 is further used for:

[0197] Determine the spatial location and spatial orientation information of the object to be identified;

[0198] In response to the sample pose information satisfying the fourth preset condition, the sample control signal is generated based on the initial neural network processing of spatial position information and spatial pose information.

[0199] In response to the sample posture information not meeting the fourth preset condition, the reachable space range and joint adjustment range of the second robotic arm are determined, and the spatial position information and spatial posture information of the object to be identified are adjusted according to the reachable space range and joint adjustment range. The second robotic arm is connected to the object to be identified, and the second robotic arm and the object to be identified together constitute the second electronic device.

[0200] It should be noted that the foregoing explanation of the camera control method also applies to the camera control device of this embodiment, and will not be repeated here.

[0201] In this embodiment, a sample data set of an initial neural network model is obtained, wherein the initial neural network model is used to generate control signals to drive the first robotic arm. The initial neural network model is trained based on the sample data set to obtain a target neural network model. Based on the target neural network model, a target control signal is generated, wherein the target control signal is used to drive the first robotic arm to control the camera to acquire target image data of the object to be identified. Thus, a target control signal for the first robotic arm can be generated quickly and accurately based on the target neural network model, enabling the camera to acquire target image data from a better perspective, thereby ensuring the recognition effect of the object to be identified based on the target image data.

[0202] Figure 8 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Figure 8 The electronic device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0203] like Figure 8 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0204] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0205] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0206] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 8 Not shown; usually referred to as a "hard drive".

[0207] although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0208] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0209] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable human interaction with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0210] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the camera control method mentioned in the foregoing embodiments.

[0211] To implement the above embodiments, this disclosure also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the camera control method proposed in the foregoing embodiments of this disclosure.

[0212] To implement the above embodiments, this disclosure also proposes a computer program product that, when the instruction processor in the computer program product is executed, performs the camera control method as proposed in the foregoing embodiments of this disclosure.

[0213] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0214] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0215] It should be noted that in the description of this disclosure, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0216] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0217] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0218] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0219] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0220] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0221] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0222] Although embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A method for controlling a camera, characterized in that, The method is executed by a first electronic device, and the first electronic device at least includes a first mechanical arm and a camera connected to the first mechanical arm, and the method comprises the following steps: obtaining a sample data set of an initial neural network model, wherein the initial neural network model is used to generate a control signal for driving the first mechanical arm; training the initial neural network model according to the sample data set to obtain a target neural network model; generating a target control signal according to the target neural network model, wherein the target control signal is used to drive the first mechanical arm to control the camera to obtain target image data of an object to be recognized; the step of obtaining the sample data set of the initial neural network model comprises: determining a control period of the initial neural network model; obtaining sample data of the initial neural network model according to the control period, wherein the sample data comprises: obtaining an initial state value of the first electronic device in an initial state according to the control period; generating a sample control signal corresponding to the initial state value based on the initial neural network; determining joint adjustment data of the first mechanical arm according to the sample control signal; driving the first mechanical arm based on the joint adjustment data to make the first electronic device in a sample state; determining a sample state value and a reward value corresponding to the joint adjustment data, wherein the initial state value, the joint adjustment data, the sample state value and the reward value are jointly used as the sample data; in response to the sample data satisfying a first preset condition, constructing the sample data set according to at least one sample data; wherein the first preset condition comprises any one of the following: the number of sample data is greater than or equal to a first preset value; the reward value of the sample data is greater than or equal to a second preset value.

2. The method of claim 1, wherein, the first mechanical arm comprises at least one joint; wherein the step of obtaining an initial state value of the first electronic device in an initial state according to the control period comprises: obtaining initial angle data of the at least one joint in the initial state; generating a first vector according to the initial angle data; obtaining initial image data of the object to be recognized based on the camera in the initial state; generating a second vector according to the initial image data; determining the initial state value according to the first vector and the second vector.

3. The method of claim 2, wherein, the step of driving the first mechanical arm based on the joint adjustment data to make the first electronic device in a sample state comprises: determining trajectory information of the at least one joint in a joint adjustment process according to the joint adjustment data; obtaining point cloud data of a scene in which the first electronic device is located; generating a collision detection result corresponding to the joint adjustment data according to the trajectory information and the point cloud data; in response to the collision detection result satisfying a second preset condition, driving the first mechanical arm based on the joint adjustment data to make the first electronic device in the sample state, wherein the second preset condition refers to a restrictive condition configured in advance for the collision detection result.

4. The method of claim 3, wherein, The determining the sample state value and the reward value corresponding to the joint adjustment data comprises: obtaining sample angle data of the at least one joint in the sample state; generating a third vector according to the sample angle data; obtaining sample image data of the to-be-recognized object based on the camera in the sample state; generating a fourth vector according to the sample image data; determining the sample state value according to the third vector and the fourth vector; determining the reward value according to the collision detection result.

5. The method of claim 4, wherein, The determining the reward value according to the collision detection result comprises: in response to the collision detection result satisfying the second preset condition, obtaining a joint adjustment speed value and an object recognition error value corresponding to the joint adjustment data, and determining the reward value according to the joint adjustment speed value and the object recognition error value; in response to the collision detection result not satisfying the second preset condition, taking a third preset value as the reward value.

6. The method of claim 5, wherein, The obtaining the joint adjustment speed value and the object recognition error value corresponding to the joint adjustment data, and determining the reward value according to the joint adjustment speed value and the object recognition error value, comprises: determining the joint adjustment speed value according to the control period and the joint adjustment data; obtaining an actual recognition value of the to-be-recognized object; obtaining a sample recognition value corresponding to the joint adjustment data; determining the object recognition error value according to the actual recognition value and the sample recognition value; in response to the object recognition error value satisfying a third preset condition, obtaining an operation result of the joint adjustment speed value and the object recognition error value based on a pre-designed calculation template as the reward value; in response to the object recognition error value not satisfying the third preset condition, taking a fourth preset value as the reward value; wherein the third preset condition refers to a threshold condition configured in advance for the object recognition error value.

7. The method of claim 1, wherein, The generating the sample control signal corresponding to the initial state value based on the initial neural network comprises: determining spatial position information and spatial posture information of the to-be-recognized object; in response to the sample posture information satisfying a fourth preset condition, processing the spatial position information and the spatial posture information based on the initial neural network to generate the sample control signal; in response to the sample posture information not satisfying the fourth preset condition, determining a reachable space range and a joint adjustment range of a second mechanical arm, and adjusting the spatial position information and the spatial posture information of the to-be-recognized object according to the reachable space range and the joint adjustment range, wherein the second mechanical arm is connected with the to-be-recognized object, and the second mechanical arm and the to-be-recognized object together constitute a second electronic device, and wherein the fourth preset condition refers to a condition configured for the sample posture information of the to-be-recognized object.

Images