Lunar miniature robot photographing pose optimization method and device

Abstract

The invention provides a lunar miniature robot photographing pose optimization method and device, and relates to the field of lunar exploration, and the method comprises the steps: determining the current image features of a real-time RGB image of a to-be-photographed target according to the real-time RGB image; determining whether the current image feature and a preset expected image feature meet a similarity requirement; if not, determining a decision action for changing the photographing pose according to the current image feature and the expected image feature; and adjusting the lunar miniature robot according to the decision action until the current image feature meets the requirement. The scheme does not need to depend on accurate positioning information, the photographing pose of the robot is optimized only based on the RGB image, and the robot can be automatically adapted to the environment in the unknown lunar surface environment without ground remote control.

Description

Technical Field

[0001] The present invention relates to the field of lunar exploration technology, and in particular to a method and device for optimizing the photographing posture of a lunar micro-robot. Background Art

[0002] With the successful launch of lunar, Martian and asteroid probes, extraterrestrial exploration technology has made initial progress. During extraterrestrial exploration, the camera installed on the robot is used to take appropriate photos of cooperative targets in extraterrestrial scenes and transmit them back to Earth. This is of great significance for understanding the extraterrestrial environment and leaving evidence of human exploration of the extraterrestrial world.

[0003] In the related art, when using a tiny lunar robot for shooting, it is often difficult to obtain the best image of the target to be photographed due to limitations of environmental conditions and technical equipment, which leads to mission failure.

[0004] Based on this, there is an urgent need for a method and device for optimizing the photography posture of a lunar micro-robot to solve the above-mentioned technical problems. Summary of the invention

[0005] The embodiment of the present invention provides a method and device for optimizing the photographing posture of a lunar micro-robot, which can autonomously adjust to the optimal photographing posture to obtain the best image that meets the task requirements.

[0006] In a first aspect, an embodiment of the present invention provides a method for optimizing the posture of a lunar micro-robot for taking photos, comprising:

[0007] Determining the current image features of the image according to the real-time RGB image of the target to be photographed;

[0008] Determining whether the current image feature and the preset expected image feature meet similarity requirements;

[0009] If not, a decision action to change the photographing posture is determined according to the current image features and the expected image features; and the lunar micro-robot is adjusted according to the decision action until the current image features meet the requirements.

[0010] In a second aspect, an embodiment of the present invention further provides a lunar micro-robot photographing posture optimization device, comprising:

[0011] A first determination module is used to determine the current image features of the image according to the real-time RGB image of the target to be photographed;

[0012] A judgment module, used to determine whether the current image feature and the preset expected image feature meet the similarity requirement;

[0013] The second determination module is used to determine the decision action of changing the photographing posture according to the current image features and the expected image features when the current image features do not meet the similarity requirements; and adjust the lunar micro-robot according to the decision action until the current image features meet the requirements.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in any embodiment of this specification is implemented.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute the method described in any embodiment of this specification.

[0016] The embodiment of the present invention provides a method and device for optimizing the photo posture of a small robot on the lunar surface. According to pure visual information, the difference between the current posture and the optimal posture is judged through intelligent perception and intelligent decision-making algorithms, so as to optimize the photo posture of the small robot and autonomously adjust it from the initial posture to the optimal photo posture. The method does not need to rely on precise positioning information, but only optimizes the photo posture of the robot based on RGB images. It adopts an intelligent control algorithm with high computational efficiency. In the unknown environment on the lunar surface, it can autonomously adapt to the environment without ground remote control. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is a flow chart of a method for optimizing the posture of a lunar micro-robot for taking photos provided by an embodiment of the present invention;

[0019] Figure 2 is a hardware architecture diagram of an electronic device provided by an embodiment of the present invention;

[0020] Figure 3 It is a structural diagram of a lunar micro-robot photographing posture optimization device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0022] As mentioned earlier, in the lunar scene, first of all, since the motion state of the lunar micro-robot varies greatly under different terrain and soil characteristics, the precise positioning method based on wheel odometer is affected, which in turn causes the target to be photographed to be out of the camera's field of view. At the same time, the limited sensors carried by the micro-robot further increase the difficulty of shooting.

[0023] Based on this, the idea of ​​the present invention is to use only the camera installed on the robot to autonomously adjust the robot's shooting posture according to pure visual information, and to optimize the shooting posture of the shooting target in a human-like manner, so as to capture a photo in which the target is completely visible, of appropriate size, and centered in the image.

[0024] The specific implementation of the above concept is described below.

[0025] Please refer to Figure 1 , an embodiment of the present invention provides a method for optimizing the posture of a lunar micro-robot for taking photos, the method comprising:

[0026] Step 100, determining the current image features of the image according to the real-time RGB image of the target to be photographed;

[0027] Step 102, determining whether the current image feature and the preset expected image feature meet similarity requirements;

[0028] Step 104: If not, determine a decision action to change the photographing posture according to the current image features and the expected image features; and adjust the lunar micro-robot according to the decision action until the current image features meet the requirements.

[0029] In the embodiment of the present invention, the difference between the current posture and the optimal posture is determined by intelligent perception and intelligent decision-making algorithms based on pure visual information, so as to optimize the shooting posture of the small robot and autonomously adjust it from the initial posture to the optimal shooting posture. This method does not need to rely on precise positioning information, but only optimizes the shooting posture of the robot based on RGB images. It adopts an intelligent control algorithm with high computational efficiency. In the unknown environment of the moon, it can autonomously adapt to the environment without ground remote control.

[0030] Described below Figure 1 How the various steps are performed.

[0031] First, with respect to step 100, the current image features of the image are determined according to the real-time RGB image of the target to be photographed.

[0032] When the robot is in a stationary state, a real-time RGB image of the object to be photographed is taken using a visible light camera installed on the robot, and the image is used as the current image.

[0033] Furthermore, the real-time RGB image is input into a trained intelligent perception network model, and a detection result of a typical component of the target to be photographed is output; the detection result includes the coordinates of the upper left vertex of a rectangular detection frame of the typical component in an image pixel coordinate system, and the width and height of the rectangular detection frame;

[0034] According to the detection result, the pixel coordinates of the center point of the rectangular detection frame are calculated:

[0035]

[0036] Among them, (x in ,y in ) is the pixel coordinate of the center point of the nth rectangular detection box in the ith real-time RGB image; (x lin ,y lin ) is the pixel coordinate of the upper left vertex of the nth rectangular detection box; w in is the width of the rectangular detection frame; h in is the height of the rectangular detection box;

[0037] According to the detection result, the total number of pixels in the rectangular detection frame is calculated:

[0038] p in =h in w in

[0039] Among them, p in is the total number of pixels in the nth rectangular detection box;

[0040] According to the pixel coordinates of the center point and the total number of pixels, the current image feature is determined as: B = [x i1 ,y i1 ,p i1 , … ,x in ,y in ,p in ].

[0041] For example, the object to be photographed is a landing assembly, and typical components include a lander, an ascender, and a national flag. In the initial image, the landing assembly is completely visible in the image, but its position is biased to the left, which does not meet the desired shooting effect. The initial image is input into the trained intelligent perception network model for detection. The detection result of the initial image detects three typical components: a lander, an ascender, and a national flag, and the number of corresponding rectangular detection frames is also three. In particular, the network model can still obtain detection results even when the assembly has very strong reflectivity.

[0042] Then, the above calculation formula can be used to calculate the pixel coordinates of the center point of each rectangular detection frame and the total number of pixels in the rectangular detection frame, so that the current image feature of the image can be obtained as follows: B = [x i1 ,y i1 ,p i1 ,…,x i3 ,y i3 ,p i3 ].

[0043] It is worth noting that if one or more typical components are not detected in the current image, the image feature vector x of the typical component in ,y in ,p in Both are 0.

[0044] Then, for step 102, it is determined whether the current image feature and the preset expected image feature meet the similarity requirement.

[0045] In the embodiment of the present invention, the desired image features, i.e., the desired photographing effect of the assembly, specifically include the coordinates (x d ,y d ) and the total number of pixels p in the rectangular frame d , the feature vector of the expected image is A = [x d1 ,y d1 ,p d1 ,…,x d3 ,y d3 ,p d3 ].

[0046] That is to say, the expected image features are calculated from images taken using the camera on the small robot in a ground simulation test field. For example, when shooting, the small robot is 6 meters away from the assembly, and the assembly is completely visible, of appropriate size and centered in the image, which is the expected image shooting effect.

[0047] After obtaining the current image features, we can compare them with the expected image features to determine the similarity between the current real-time image and the expected image.

[0048] The comparison process includes: calculating the similarity value between the real-time RGB image and the expected image according to the current image features and the expected image features; determining whether the similarity value is greater than a preset similarity index threshold; if so, determining that the real-time RGB image meets the similarity requirement, otherwise determining that the real-time RGB image does not meet the similarity requirement.

[0049] For step 104, when the current image features do not meet the similarity requirements, a decision action to change the photographing posture is determined based on the current image features and the expected image features; and the lunar micro-robot is adjusted according to the decision action until the current image features meet the requirements.

[0050] Once it is determined that the current image does not meet the requirements of the desired image, it is necessary to adjust the shooting posture of the lunar microrobot to obtain an image that better meets the requirements.

[0051] First, the current image features and the expected image features need to be merged. For example, if the expected image features are A = [x d1 ,y d1 ,p d1 ,…,x dn ,y dn ,p dn ], the current image feature is B = [x i1 ,y i1 ,p i1 ,…,x in ,y in ,p in ], the merged feature vector obtained is C=[A,B]. Then the feature vector C is input into the trained intelligent decision network model, and the decision action is output. The robot can adjust its movement according to the decision action.

[0052] In an embodiment of the present invention, the intelligent decision network model is obtained by training a preset action decision network and a value network using a training sample set, and the training sample set includes a training target image obtained by the robot at any first position of the target to be photographed, and a training initial image obtained by the robot at any second position within a circular area with the first position as the center.

[0053] The training sample set is input into the action decision network with set state space and action space for training, so as to realize continuous optimization of action decision until the intelligent decision network model is obtained.

[0054] Specifically, first, an 18-dimensional state space is established based on the merged state vector, and then an action space is established based on the decision-making actions that the lunar micro-robot needs to perform.

[0055] In an embodiment of the present invention, decision actions include forward movement, backward movement, left front movement, right front movement, left rear movement, and right rear movement, and the corresponding decision instructions are 0-5 respectively. For example, the decision instruction output by the intelligent decision network is 2, which means that the small robot needs to move to the left front direction.

[0056] Furthermore, when the decision action is forward movement or backward movement, the robot is controlled to perform linear movement according to the instruction; when the decision action is any one of left front movement, right front movement, left rear movement and right rear movement, the robot is controlled to perform curved movement according to the instruction.

[0057] For example, the forward and backward movements move 20 cm according to the instructions, and the left front movement, right front movement, left rear movement, and right rear movement move 20 cm in arc length in the corresponding direction according to the instructions. The angle corresponding to the arc length is 15°. The small robot moves to the left front direction according to the decision instruction in step 6.

[0058] It is worth noting that the distances and angles in the above embodiments are ideal conditions. Under different soil and robot wheel characteristics, the actual movement distance and angle may not be consistent with the ideal conditions, and the present invention can also adapt to this.

[0059] Then set the reward function of the value network. The reward function is used to evaluate the reward score of the action output by the action decision network. The reward function includes visibility reward, task completion reward, total step reward and single-step similarity reward. The visibility reward is used to characterize whether the latest image taken after moving according to the action output by the action decision network contains the target to be photographed; the task completion reward is used to characterize whether the latest image meets the shooting requirements; the total step reward is used to characterize the running step reward in a round of training; the single-step similarity reward is used to characterize whether the current image taken after completing an action is more similar to the training target image than the previous image.

[0060] The action decision network will make action decisions based on the merged feature vectors of the initial training image and the training target image. The robot will perform corresponding actions based on the action decisions to make the latest captured image more similar to the training target image. A reward curve will be constructed based on the reward score. When the curve meets the convergence condition, the network will end training and obtain an intelligent decision network model.

[0061] In an embodiment of the present invention, after the robot adjusts its position according to the decision result of the above-mentioned intelligent decision-making network model, it needs to take a new photo and compare it with the expected image. If it meets the requirements, the shooting is terminated. If it does not meet the requirements, the above-mentioned adjustment process is repeated until the image meets the requirements. In particular, if the captured image still does not meet the expectations after the number of repetitions reaches the preset adjustment upper limit, the one with the highest similarity is selected from all the captured images as the best image, and the shooting is terminated.

[0062] like Figure 2 , Figure 3 As shown, the embodiment of the present invention provides a lunar micro-robot photography posture optimization device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From the hardware level, Figure 2 As shown in the figure, it is a hardware architecture diagram of an electronic device in which a lunar surface micro robot photography posture optimization device is provided in an embodiment of the present invention. Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in the embodiment may also include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, Figure 3 As shown, as a device in a logical sense, the CPU of the electronic device in which it is located reads the corresponding computer program in the non-volatile memory into the memory and runs it. This embodiment provides a lunar micro-robot photography posture optimization device, including:

[0063] A first determination module 300 is used to determine the current image features of the image according to the real-time RGB image of the target to be photographed;

[0064] A judgment module 302 is used to determine whether the current image feature and the preset expected image feature meet the similarity requirement;

[0065] The second determination module 304 is used to determine a decision action to change the photographing posture according to the current image features and the expected image features when the current image features do not meet the similarity requirements; and adjust the lunar micro-robot according to the decision action until the current image features meet the requirements.

[0066] In the embodiment of the present invention, when the first determination module 300 determines the current image features of the image according to the real-time RGB image of the target to be photographed, it is specifically used to perform the following operations:

[0067] Input the real-time RGB image into the trained intelligent perception network model, and output the detection result of the typical component of the target to be photographed; the detection result includes the coordinates of the upper left vertex of the rectangular detection box of the typical component in the image pixel coordinate system, and the width and height of the rectangular detection box;

[0068] According to the detection result, the pixel coordinates of the center point of the rectangular detection frame are calculated:

[0069]

[0070] Among them, (x in ,y in ) is the pixel coordinate of the center point of the nth rectangular detection box in the ith real-time RGB image; (x lin ,y lin ) is the pixel coordinate of the upper left vertex of the nth rectangular detection box; w in is the width of the rectangular detection frame; h in is the height of the rectangular detection box;

[0071] According to the detection result, the total number of pixels in the rectangular detection frame is calculated:

[0072] p in =h in w in

[0073] Among them, p in is the total number of pixels in the nth rectangular detection box;

[0074] According to the pixel coordinates of the center point and the total number of pixels, the current image feature is determined as: B = [x i1 ,y i1 ,p i1 , … ,x in ,y in ,p in ].

[0075] In the embodiment of the present invention, when determining whether the current image feature and the preset expected image feature meet the similarity requirement, the judgment module 302 is specifically used to perform the following operations: calculating the similarity value between the real-time RGB image and the expected image according to the current image feature and the expected image feature; determining whether the similarity value is greater than a preset similarity index threshold; if so, determining that the real-time RGB image meets the similarity requirement, otherwise determining that the real-time RGB image does not meet the similarity requirement.

[0076] In an embodiment of the present invention, when the second determination module 304 determines the decision action of changing the photographing posture based on the current image feature and the expected image feature when the current image feature does not meet the similarity requirement, it is specifically used to perform the following operations: merging the current image feature and the expected image feature; inputting the merged feature vector into the trained intelligent decision network model, and outputting the decision action.

[0077] In the embodiment of the present invention, the decision actions include forward movement, backward movement, left front movement, right front movement, left rear movement and right rear movement.

[0078] In an embodiment of the present invention, the second determination module 304 is specifically used to perform the following operations when adjusting the lunar micro-robot according to the decision action: when the decision action is forward movement or backward movement, the robot is controlled to move in a straight line according to the instruction; when the decision action is any one of left front movement, right front movement, left rear movement and right rear movement, the robot is controlled to move in a curve according to the instruction.

[0079] In the embodiment of the present invention, when the number of adjustments reaches a preset upper limit, the real-time RGB image with the highest similarity to the expected image is determined as the best image, and the shooting is terminated.

[0080] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a device for optimizing the posture of a lunar micro-robot taking photos. In other embodiments of the present invention, a device for optimizing the posture of a lunar micro-robot taking photos may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0081] The information interaction, execution process and other contents between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For the specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0082] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a method for optimizing the photographing posture of a lunar micro-robot in any embodiment of the present invention is implemented.

[0083] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes a method for optimizing the photographing posture of a lunar micro-robot in any embodiment of the present invention.

[0084] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0085] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0086] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0087] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0088] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0089] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical factors in the process, method, article or device including the elements.

[0090] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing the posture of a lunar micro-robot for taking photos, characterized in that: include: Determining the current image features of the image according to the real-time RGB image of the target to be photographed; Determining whether the current image feature and the preset expected image feature meet similarity requirements; If not, a decision action to change the photographing posture is determined according to the current image features and the expected image features; and the lunar micro-robot is adjusted according to the decision action until the current image features meet the requirements.

2. The method according to claim 1, characterized in that The step of determining the current image features of the image according to the real-time RGB image of the target to be photographed includes: Input the real-time RGB image into the trained intelligent perception network model, and output the detection result of the typical component of the target to be photographed; the detection result includes the coordinates of the upper left vertex of the rectangular detection box of the typical component in the image pixel coordinate system, and the width and height of the rectangular detection box; According to the detection result, the pixel coordinates of the center point of the rectangular detection frame are calculated: Among them, (x in ,y in ) is the pixel coordinate of the center point of the nth rectangular detection box in the ith real-time RGB image; (x lin ,y lin ) is the pixel coordinate of the upper left vertex of the nth rectangular detection box; w in is the width of the rectangular detection frame; h in is the height of the rectangular detection box; According to the detection result, the total number of pixels in the rectangular detection frame is calculated: p in =h in w in Among them, p in is the total number of pixels in the nth rectangular detection box; According to the pixel coordinates of the center point and the total number of pixels, the current image feature is determined as: B = [x i1 ,y i1 ,p i1 , … ,x in ,y in ,p in ].

3. The method according to claim 1, characterized in that The determining whether the current image feature and the preset expected image feature meet similarity requirements includes: Calculating a similarity value between the real-time RGB image and the expected image according to the current image feature and the expected image feature; Determining whether the similarity value is greater than a preset similarity index threshold; If so, it is determined that the real-time RGB image meets the similarity requirement; otherwise, it is determined that the real-time RGB image does not meet the similarity requirement.

4. The method according to claim 1, characterized in that: The step of determining a decision action for changing the photographing posture according to the current image feature and the expected image feature includes: Combining the current image feature and the expected image feature; The combined feature vector is input into the trained intelligent decision network model, and the decision action is obtained as output.

5. The method according to claim 4, characterized in that The decision actions include forward movement, backward movement, left front movement, right front movement, left rear movement and right rear movement.

6. The method according to claim 5, characterized in that The step of adjusting the lunar surface micro-robot according to the decision action comprises: When the decision action is forward movement or backward movement, the robot is controlled to move in a straight line according to the instruction; When the decision action is any one of left front movement, right front movement, left rear movement and right rear movement, the robot is controlled to perform curved movement according to the instruction.

7. The method according to claim 3, characterized in that After adjusting the lunar surface micro-robot according to the decision action until the current image feature meets the requirements, it also includes: When the number of adjustments reaches a preset upper limit, the real-time RGB image with the highest similarity to the expected image is determined as the best image, and the shooting is terminated.

8. A lunar micro-robot photography posture optimization device, characterized in that: include: A first determination module is used to determine the current image features of the image according to the real-time RGB image of the target to be photographed; A judgment module, used to determine whether the current image feature and the preset expected image feature meet the similarity requirement; The second determination module is used to determine the decision action of changing the photographing posture according to the current image features and the expected image features when the current image features do not meet the similarity requirements; and adjust the lunar micro-robot according to the decision action until the current image features meet the requirements.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images