Virtual perspective image construction method and device, control equipment and readable storage medium

Abstract

The application provides a virtual perspective image construction method and device, a control device and a readable storage medium, and relates to the technical field of computer vision. After determining the stereoscopic depth distribution graph between a target robot and a target scene based on a real binocular field photographed by the target robot at a current real perspective, the application converts the stereoscopic depth distribution graph to obtain a virtual binocular parallax graph according to the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual perspective, then performs image offset on the real binocular scene graph according to the virtual binocular parallax graph to obtain an initial virtual binocular scene graph, and further repairs the initial virtual binocular scene graph according to the stereoscopic depth distribution graph to obtain a target virtual binocular scene graph, so that the binocular scene image with good image texture consistency at different perspectives can be quickly constructed without adjusting the real camera perspective, and the binocular training image set can be constructed at low cost and quickly.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method and apparatus for constructing virtual perspective images, a control device, and a readable storage medium. Background Technology

[0002] With the continuous development of science and technology, computer vision technology has received widespread attention from various industries due to its significant research and application value, and is used to realize a variety of business functions (such as robot obstacle avoidance, 3D video construction, and 3D model construction). In practical applications, these business functions often require the use of a large number of training image sets to train a neural network model. The sufficiency of the training image set data will significantly affect the performance of the neural network model.

[0003] For binocular robots, the required functionalities often need to be implemented based on binocular scene images captured by their own binocular cameras. These functionalities also require a large set of binocular training images for model training. However, it's important to note that a binocular robot typically uses only one binocular camera configuration (i.e., a fixed viewing angle), while the binocular training image set needs to consist of images from different viewing angles. Constructing a binocular training image set through binocular image acquisition requires significant costs to adjust the robot's viewing angle and, after each adjustment, to perform epipolar alignment image correction on the captured binocular scene images to obtain the corresponding binocular training images from the robot's viewing angle. Therefore, this image acquisition-based training set construction method cannot quickly build the desired binocular training image set, and the cost of constructing the training image set is extremely high. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a virtual perspective image construction method and apparatus, control device and readable storage medium, which can quickly construct binocular scene images with good image texture consistency of the robot under different perspectives without adjusting the position configuration of the robot's real camera, so as to quickly construct the desired binocular training image set and reduce the cost of constructing the training image set.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0006] In a first aspect, this application provides a method for constructing a virtual perspective image, the method comprising:

[0007] Acquire a real binocular scene image of the target robot from its current real-world perspective;

[0008] Determine the stereo depth distribution map between the target robot and the target scene based on the real binocular scene map;

[0009] Based on the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual viewpoint, the stereo depth distribution map is subjected to depth parallax transformation to obtain a virtual binocular parallax map of the target robot for the target scene under the target virtual viewpoint.

[0010] Based on the virtual binocular disparity map, the real binocular scene map is image offset to obtain an initial virtual binocular scene map that matches the target virtual viewpoint;

[0011] The initial virtual binocular scene image is inpainted based on the stereo depth distribution map to obtain the target virtual binocular scene image.

[0012] In an optional implementation, the real binocular scene map includes a real left-eye scene map and a real right-eye scene map. The step of determining the stereo depth distribution map between the target robot and the target scene based on the real binocular scene map includes:

[0013] Feature extraction is performed on the real left-eye scene image and the real right-eye scene image respectively to obtain the corresponding left-eye scene feature image and right-eye scene feature image;

[0014] Determine the disparity prediction distribution map of each of the pre-stored disparity values ​​between the left-eye scene feature map and the right-eye scene feature map;

[0015] The image is stitched together from the disparity prediction distribution maps corresponding to each disparity value, and the pre-stored disparity regression model is called to perform disparity regression on the stitched binocular disparity distribution map to obtain the real binocular disparity map corresponding to the current real viewpoint.

[0016] Based on the current camera focal length of the target robot and the actual camera baseline distance corresponding to the current actual viewpoint, the actual binocular disparity map is subjected to disparity depth transformation to obtain the stereo depth distribution map.

[0017] In an optional implementation, the step of determining the disparity prediction distribution map of each of the pre-stored multiple disparity values ​​between the left-eye scene feature map and the right-eye scene feature map includes:

[0018] For each preset disparity value, the right eye scene feature map is shifted according to the disparity value in the feature extraction window of the right eye scene feature map to obtain the corresponding shifted feature map;

[0019] The offset feature map corresponding to this disparity value is stitched together with the real left-eye scene image to obtain a binocular disparity feature map corresponding to this disparity value.

[0020] The pre-stored disparity matching network model based on the Transformer model is invoked to perform pixel-level feature matching on the binocular disparity feature map corresponding to the disparity value, thereby obtaining the disparity prediction distribution map corresponding to the disparity value.

[0021] In an optional implementation, the step of calling a pre-stored disparity regression model to perform disparity regression on the stitched binocular disparity distribution map to obtain a true binocular disparity map corresponding to the current true viewpoint includes:

[0022] The binocular disparity distribution map is input into the disparity regression model for disparity distribution redistribution processing to obtain all possible disparity values ​​and their respective probabilities of occurrence at each pixel in the real binocular disparity map.

[0023] For each pixel in the true binocular disparity map, a weighted summation is performed on all possible disparity values ​​corresponding to that pixel and the probability of occurrence of each of the possible disparity values ​​to obtain the true disparity value of the true binocular disparity map at that pixel.

[0024] In an optional implementation, the disparity-depth correlation required to perform a disparity-depth conversion operation or a depth-disparity conversion operation is expressed by the following formula:

[0025]

[0026] Where D represents the parallax value between the subject and the left and right eye cameras of the target robot, which are on the same baseline; b represents the camera baseline distance between the left and right eye cameras of the target robot; f represents the camera focal length of the target robot; and Z represents the stereo depth value between the subject and the target robot.

[0027] In an optional implementation, the initial virtual binocular scene image includes an initial virtual left-eye scene image, and the target virtual binocular scene image includes a target virtual left-eye scene image. The step of performing image inpainting on the initial virtual left-eye scene image based on the stereo depth distribution map to obtain the target virtual left-eye scene image includes:

[0028] The stereo depth value of each effective pixel in the initial virtual left eye scene image is determined based on the stereo depth distribution map.

[0029] For each first pixel to be repaired in the initial virtual left eye scene image, the left eye repair template is called to determine multiple first valid reference pixels in the initial virtual left eye scene image that are located around the first pixel to be repaired.

[0030] Based on the stereo depth value and pixel position of each effective pixel in the initial virtual left eye scene image, a plurality of first calibration pixels are selected in the initial virtual left eye scene image whose relative position distribution, depth level and color features are consistent with the plurality of first effective reference pixels. If the absolute value of the difference between the stereo depth values ​​of two effective pixels is greater than a preset spacing threshold, then the two effective pixels are at different depth levels.

[0031] The left eye repair template is invoked to find the first effective substitute pixel within the encirclement of the plurality of first calibrated pixels in the initial virtual left eye scene image, and the first effective substitute pixel is used to perform pixel replacement processing on the first pixel to be repaired to obtain the effective pixel corresponding to the first pixel to be repaired.

[0032] The initial virtual left eye scene image after completing the pixel replacement operation of all the first pixels to be repaired is used as the target virtual left eye scene image.

[0033] In an optional implementation, the initial virtual binocular scene image further includes an initial virtual right-eye scene image, and the target virtual binocular scene image further includes a target virtual right-eye scene image. The step of performing image inpainting on the initial virtual right-eye scene image based on the stereo depth distribution map to obtain the target virtual right-eye scene image includes:

[0034] The stereo depth value of each effective pixel in the initial virtual right eye scene image is determined based on the stereo depth distribution map.

[0035] For each second pixel to be repaired in the initial virtual right eye scene image, the right eye repair template is called to determine multiple second valid reference pixels in the initial virtual right eye scene image that are located around the second pixel to be repaired.

[0036] Based on the stereo depth value and pixel position of each effective pixel in the initial virtual right eye scene image, a plurality of second calibration pixels are selected in the initial virtual right eye scene image whose relative position distribution, depth level and color features are consistent with the plurality of second effective reference pixels.

[0037] The right eye repair template is invoked to find the second effective substitute pixel within the encirclement of the plurality of second calibration pixels in the initial virtual right eye scene image, and the second effective substitute pixel is used to perform pixel replacement processing on the second pixel to be repaired to obtain the effective pixel corresponding to the second pixel to be repaired.

[0038] The initial virtual right eye scene image after completing the pixel replacement operation for all second pixels to be repaired is used as the target virtual right eye scene image.

[0039] Secondly, this application provides a virtual perspective image construction apparatus, the apparatus comprising:

[0040] The real scene image acquisition module is used to acquire real binocular scene images of the target scene captured by the target robot from the current real viewpoint.

[0041] A stereo depth map determination module is used to determine a stereo depth distribution map between the target robot and the target scene based on the real binocular scene map;

[0042] The virtual disparity map conversion module is used to perform depth disparity conversion on the stereo depth distribution map based on the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual viewpoint, so as to obtain a virtual binocular disparity map of the target robot for the target scene under the target virtual viewpoint.

[0043] The virtual scene graph construction module is used to perform image offset on the real binocular scene graph based on the virtual binocular disparity graph to obtain an initial virtual binocular scene graph that matches the target virtual viewpoint;

[0044] The virtual scene image restoration module is used to perform image restoration on the initial virtual binocular scene image based on the stereo depth distribution map to obtain the target virtual binocular scene image.

[0045] Thirdly, this application provides a control device, including a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the virtual perspective image construction method described in any of the foregoing embodiments.

[0046] Fourthly, this application provides a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the virtual perspective image construction method described in any of the foregoing embodiments.

[0047] In this case, the beneficial effects of the embodiments of this application may include the following:

[0048] This application determines the stereo depth distribution map between the target robot and the target scene based on the real binocular scene image captured by the target robot from its current real-world perspective. Then, according to the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual perspective, the stereo depth distribution map is transformed into a depth parallax map to obtain a virtual binocular parallax map of the target robot for the target scene from the target virtual perspective. Next, image offset is performed on the real binocular scene image based on the virtual binocular parallax map to obtain an initial virtual binocular scene image that matches the target virtual perspective. Then, image inpainting is performed on the initial virtual binocular scene image based on the stereo depth distribution map to obtain a target virtual binocular scene image with good image texture consistency. Thus, without adjusting the robot's real camera position configuration, binocular scene images with good image texture consistency of the corresponding robot under different perspectives can be quickly constructed to rapidly build the desired binocular training image set and reduce the cost of building the training image set.

[0049] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of the composition of the control device provided in the embodiments of this application;

[0052] Figure 2 A flowchart illustrating the virtual perspective image construction method provided in this application embodiment;

[0053] Figure 3 for Figure 2 A flowchart illustrating the sub-steps included in step S220;

[0054] Figure 4 for Figure 3 The flowchart of the sub-steps included in sub-step S222 is shown below;

[0055] Figure 5 for Figure 3 A flowchart illustrating the sub-steps included in sub-step S223;

[0056] Figure 6 This application provides an illustration of the effect of parallax depth correlation in its embodiments.

[0057] Figure 7 for Figure 2 One of the flowcharts for the sub-steps included in step S250;

[0058] Figure 8 for Figure 2 The second flowchart illustrates the sub-steps included in step S250.

[0059] Figure 9 for Figure 2 A schematic diagram illustrating the image repair effect of step S250;

[0060] Figure 10 This is a schematic diagram of the composition of the virtual perspective image construction apparatus provided in the embodiments of this application.

[0061] Icons: 10-Control device; 11-Memory; 12-Processor; 13-Communication unit; 100-Virtual viewpoint image construction device; 110-Real scene map acquisition module; 120-Stereo depth map determination module; 130-Virtual parallax map conversion module; 140-Virtual scene map construction module; 150-Virtual scene map repair module. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0064] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0065] In the description of this application, it should be understood that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this application based on the specific circumstances.

[0066] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0067] Please refer to Figure 1 , Figure 1 yes Figure 1 This is a schematic diagram of the control device 10 provided in this embodiment. In this embodiment, the control device 10 can communicate with at least one binocular robot and control the at least one binocular robot to acquire images of the current scene from its own real camera viewpoint (i.e., maintaining the real camera baseline distance state). This allows the control device 10 to simulate and construct virtual binocular scene images of the corresponding binocular robot from different viewpoints based on the real binocular scene images acquired by each binocular robot, thereby quickly constructing the desired binocular training image set and effectively reducing the cost of constructing the training image set. The real camera viewpoints of different binocular robots can be the same or different; the scenes in which different binocular robots are located can be the same or different.

[0068] In addition, the control device 10 can also be deployed independently on a single binocular robot, so that it can only control the binocular robot where the control device 10 is located, so as to quickly construct binocular scene images with good image texture consistency of the binocular robot under different perspectives.

[0069] In this embodiment, the control device 10 may include a memory 11, a processor 12, a communication unit 13, and a virtual perspective image construction device 100. The memory 11, the processor 12, and the communication unit 13 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0070] In this embodiment, the memory 11 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The memory 11 is used to store computer programs, and the processor 12 can execute the computer programs accordingly after receiving execution instructions.

[0071] In this embodiment, the processor 12 can be an integrated circuit chip with signal processing capabilities. The processor 12 can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0072] In this embodiment, the communication unit 13 is used to establish a communication connection between the control device 10 and one or more binocular robots via a network, and to send and receive data through the network, wherein the network includes wired communication networks and wireless communication networks. For example, the control device 10 can control the corresponding binocular robot to perform image acquisition operations through the communication unit 13.

[0073] In this embodiment, the virtual perspective image construction device 100 includes at least one software functional module that can be stored in the memory 11 in the form of software or firmware or embedded in the operating system of the control device 10. The processor 12 can be used to execute the executable modules stored in the memory 11, such as the software functional modules and computer programs included in the virtual perspective image construction device 100. The control device 10, through the virtual perspective image construction device 100, can quickly simulate and construct binocular scene images with good image texture consistency for the robot under different perspectives without adjusting the robot's actual camera position configuration, thereby rapidly constructing the desired binocular training image set and reducing the cost of constructing the training image set.

[0074] Understandable, Figure 1 The block diagram shown is only a schematic diagram of one configuration of the control device 10. The control device 10 may also include a... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown. Figure 1 The components shown can be implemented using hardware, software, or a combination thereof.

[0075] In this application, to ensure that the control device 10 can quickly construct binocular scene images with good image texture consistency from different viewpoints without adjusting the robot's actual camera position configuration, thereby rapidly constructing the desired binocular training image set and reducing the cost of constructing the training image set, this application provides a virtual viewpoint image construction method to achieve the aforementioned objective. The virtual viewpoint image construction method provided in this application will be described in detail below.

[0076] Please refer to Figure 2 , Figure 2 This is a schematic flowchart of a virtual perspective image construction method provided in an embodiment of this application. In this embodiment, the virtual perspective image construction method may include steps S210 to S250.

[0077] Step S210: Obtain a real binocular scene image of the target scene captured by the target robot from the current real perspective.

[0078] In this embodiment, the real binocular scene image includes a real left-eye scene image and a real right-eye scene image. The control device 10 can control the target robot to go to the target scene and control the target robot to use its own binocular camera to acquire images of the target scene at the current real viewpoint (i.e., the current real camera baseline distance) to obtain the real binocular scene image.

[0079] Step S220: Determine the stereo depth distribution map between the target robot and the target scene based on the real binocular scene map.

[0080] In this embodiment, after obtaining a true binocular scene map matching the current true viewpoint of the target robot, the control device 10 can determine the binocular disparity map between the true right-eye scene map and the true left-eye scene map by calling a pre-stored cost volume-based disparity construction neural network model. Then, combining the target robot's current camera focal length and the true camera baseline distance corresponding to the current true viewpoint, the device performs disparity depth transformation on the binocular disparity map to obtain the stereo depth distribution map. The stereo depth distribution map is used to characterize the specific distribution of distance information between the target robot and various objects within the target scene.

[0081] It is worth noting that building neural network models based on cost volume disparity often requires a large amount of memory and consumes a lot of computing resources to calculate the final result, resulting in high computational complexity and high memory consumption.

[0082] Therefore, this application also provides a stereo depth distribution map determination scheme with low computational complexity and low memory consumption, improving upon the problems existing in the aforementioned neural network model constructed based on cost volume disparity. The specific implementation process of the stereo depth distribution map determination scheme is as follows:

[0083] Please refer to Figure 3 , Figure 3 yes Figure 2 A flowchart illustrating the sub-steps included in step S220 is provided. In this embodiment, step S220 may include sub-steps S221 to S224.

[0084] Sub-step S221: Extract features from the real left-eye scene image and the real right-eye scene image respectively to obtain the corresponding left-eye scene feature map and right-eye scene feature map.

[0085] In this embodiment, the control device 10 can extract features from the real left-eye scene image and the real right-eye scene image from the dimensions of bath (number of image frames), channels (number of image channels), height (number of pixels in the vertical direction of the image), and width (number of pixels in the horizontal direction of the image) to obtain a right-eye scene feature map that matches the real right-eye scene image and a left-eye scene feature map that matches the real left-eye scene image.

[0086] Sub-step S222: Determine the pre-stored disparity prediction distribution map of each of the various disparity values ​​between the left-eye scene feature map and the right-eye scene feature map.

[0087] In this embodiment, the pre-stored multiple disparity values ​​are used to represent all disparity values ​​that the target robot's current real viewpoint can support. The control device 10 can construct a corresponding disparity prediction distribution map between the left-eye scene feature map and the right-eye scene feature map for each disparity value, so as to intuitively understand the distribution probability of the corresponding disparity value between the left-eye scene feature map and the right-eye scene feature map through the disparity prediction distribution map corresponding to each disparity value.

[0088] Alternatively, please refer to Figure 4 , Figure 4 yes Figure 3 The flowchart of sub-step S222 is shown below. In the embodiments of this application, sub-step S222 may include sub-steps S2221 to S2223.

[0089] Sub-step S2221: For each preset disparity value, the right eye scene feature map is shifted according to the disparity value in the feature extraction window of the right eye scene feature map to obtain the corresponding shifted feature map.

[0090] In this embodiment, when the control device 10 performs feature offset on the right-eye scene feature map according to a certain disparity value, it must ensure that the position of the feature extraction window corresponding to the right-eye scene feature map remains unchanged, while shifting the image features of the right-eye scene feature map under the feature extraction window in the horizontal direction of the image as a whole, and then performing feature extraction operation on the right-eye scene feature map after feature offset by the feature extraction window to obtain an offset feature map that matches the disparity value.

[0091] Sub-step S2222 involves stitching the offset feature map corresponding to the disparity value with the real left-eye scene image to obtain the binocular disparity feature map corresponding to the disparity value.

[0092] Sub-step S2223 calls the pre-stored disparity matching network model based on the Transformer model to perform pixel-level feature matching on the binocular disparity feature map corresponding to the disparity value, and obtains the disparity prediction distribution map corresponding to the disparity value.

[0093] Among them, the disparity matching network model based on the Transformer model can divide the input binocular disparity feature map into feature blocks, and in the fully connected layer, the channel dimension feature content of the corresponding feature block is transformed into the weight values ​​of the query dimension, key dimension, and value dimension that are interrelated. Then, the Attention mechanism is used to perform pixel-level feature matching between each feature block and other feature blocks, so as to determine the distribution probability of the corresponding disparity value between the left eye scene feature map and the right eye scene feature map (i.e., the disparity prediction distribution map corresponding to the disparity value) based on the final pixel-level feature matching result.

[0094] Therefore, by executing the above sub-steps S2221 to S2223, this application avoids storing disparity at the control device 10 to build a neural network. Instead, it considers the distribution probability of each disparity value at the real binocular scene map separately by using the idea of ​​disparity decoupling. This effectively reduces the memory consumption and computing resource consumption required to estimate the disparity distribution, and reduces the computational complexity of the entire stereo depth calculation process.

[0095] Sub-step S223 involves stitching together the disparity prediction distribution maps corresponding to each disparity value, and then calling the pre-stored disparity regression model to perform disparity regression on the stitched binocular disparity distribution map to obtain the true binocular disparity map corresponding to the current real viewpoint.

[0096] In this embodiment, the disparity regression model pre-stored in the control device 10 can be constructed using 3D convolution operations as the core, or it can be trained using the Softargmin algorithm and a smoothed L1 norm loss function. The Softargmin algorithm can effectively improve the disparity regression accuracy of the disparity regression model, so as to ensure that the corresponding disparity regression model outputs a high-precision true binocular disparity map between the true left-eye scene map and the true right-eye scene map.

[0097] Alternatively, please refer to Figure 5 , Figure 5 yes Figure 3 The flowchart of sub-step S223 is shown below. In this embodiment of the application, the disparity regression model is trained based on the Softargmin algorithm, and sub-step S223 may include sub-steps S2231 to S2232.

[0098] Sub-step S2231: Input the binocular disparity distribution map into the disparity regression model for disparity distribution redistribution processing to obtain all possible disparity values ​​and their respective probabilities at each pixel in the real binocular disparity map.

[0099] Sub-step S2232: For each pixel in the real binocular disparity map, perform a weighted summation of all possible disparity values ​​and their respective probabilities at that pixel to obtain the real disparity value of the real binocular disparity map at that pixel.

[0100] Therefore, by executing the above sub-steps S2231 to S2232, this application can comprehensively consider the estimated distribution of various pre-stored disparity values ​​at the real binocular disparity map using the Softargmin algorithm, and accurately calculate the actual distribution of various disparity values ​​substantially involved between the real left-eye scene map and the real right-eye scene map.

[0101] Sub-step S224: Based on the current camera focal length of the target robot and the distance to the real camera baseline corresponding to the current real viewpoint, perform disparity depth transformation on the real binocular disparity map to obtain a stereo depth distribution map.

[0102] In this embodiment, refer to Figure 6 The diagram shown illustrates the effect of parallax depth correlation. P represents the subject being photographed, and O... L O is used to represent the projection center of the target robot's left eye camera. R The projection center of the right eye camera of the target robot is represented by 'b', the distance between the projection centers of the left and right eye cameras (i.e., the current camera baseline distance corresponding to the current real viewpoint) is represented by 'f', the focal length provided by a single camera at the target robot is represented by 'L', and the number of imageable pixels in the horizontal direction of the camera plane of the corresponding camera (including the left and right eye cameras) is represented by 'P'. L P is used to represent the image point of the subject on the camera plane of the left-eye camera. R Z represents the image point of the subject on the camera plane of the right eye camera, Z represents the stereo depth value between the subject and the target robot, and X represents the depth value between the subject and the target robot. L Used to represent the imaging point P of the left eye camera L The distance X to the left edge of the left-eye camera's image. R Used to represent the imaging point P of the right eye camera R The distance between the left and right eye cameras' left imaging boundaries, and the parallax value of the subject between the left and right eye cameras, is |X|. L -X R At this point, the disparity depth correlation can be expressed by the following formula:

[0103]

[0104] Where D represents the parallax value between the subject and the left and right eye cameras of the target robot, which are on the same baseline; b represents the camera baseline distance between the left and right eye cameras of the target robot; f represents the camera focal length of the target robot; and Z represents the stereo depth value between the subject and the target robot.

[0105] Therefore, the control device 10 can determine the stereo depth value of each pixel in the real binocular disparity map at the stereo depth distribution map based on the above-mentioned disparity depth correlation relationship, combined with the current camera focal length of the target robot and the real camera baseline distance corresponding to the current real viewpoint. At this time, the real binocular disparity map can be converted into the stereo depth distribution map.

[0106] Therefore, by executing the above sub-steps S221 to S224, this application can effectively reduce the computational complexity and memory consumption in the process of determining the stereo depth distribution map, and improve the construction efficiency of the entire virtual perspective image construction operation.

[0107] Step S230: Based on the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual viewpoint, perform depth parallax transformation on the stereo depth distribution map to obtain the virtual binocular parallax map of the target robot for the target scene under the target virtual viewpoint.

[0108] In this embodiment, the target virtual viewpoint is the viewpoint condition to be simulated and constructed by the user, and there can be multiple such views, each of which is different from the others. At this time, the control device 10 can determine the virtual disparity value corresponding to each pixel in the stereo depth distribution map at the corresponding virtual binocular disparity map based on the expression of the disparity depth correlation described above, combined with the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual viewpoint. Then, the stereo depth distribution map can be converted into a virtual binocular disparity map corresponding to the target virtual viewpoint.

[0109] Step S240: Based on the virtual binocular disparity map, perform image offset on the real binocular scene map to obtain an initial virtual binocular scene map that matches the target virtual viewpoint.

[0110] In this embodiment, when the control device 10 calculates the virtual binocular disparity map of the target robot for the target scene from the target virtual perspective, it can select the real left-eye scene map in the real binocular scene map as a reference, and distribute the disparity values ​​in the entire virtual binocular disparity map equally to the left-eye camera and right-eye camera in the target virtual perspective for scene image reconstruction processing (i.e. image offset processing) to obtain the initial virtual left-eye scene map and the initial virtual left-eye scene map included in the initial virtual binocular scene map.

[0111] Step S250: Perform image restoration on the initial virtual binocular scene image based on the stereo depth distribution map to obtain the target virtual binocular scene image.

[0112] In this embodiment, after the control device 10 initially simulates and constructs the corresponding robot's binocular scene image from the target virtual perspective without adjusting the robot's real camera position configuration, in order to avoid obvious holes in the final output binocular scene image, the control device 10 needs to perform image repair on the initially simulated and constructed virtual binocular scene image to ensure that the target virtual binocular scene image obtained after repair has good image texture consistency.

[0113] Alternatively, please refer to Figure 7 , Figure 7 yes Figure 2 One of the flowcharts for the sub-steps included in step S250 is shown below. In this embodiment of the application, the target virtual binocular scene image may include a target virtual left eye scene image. The step of performing image restoration on the initial virtual left eye scene image according to the stereo depth distribution map in step S250 to obtain the target virtual left eye scene image may include sub-steps S251 to S255, so as to distinguish the foreground and background conditions during the restoration process of the initial virtual left eye scene image and avoid the phenomenon of blurred foreground edges due to the mixing of foreground and background content in the repaired foreground texture.

[0114] Sub-step S251: Determine the stereo depth value of each effective pixel in the initial virtual left eye scene map based on the stereo depth distribution map.

[0115] In this embodiment, the effective pixels in the initial virtual left-eye scene image are non-hole pixels with depth information. These pixels can be obtained by repairing pixels that originally belonged to holes, or they can always be non-hole pixels. The control device 10 can determine the stereo depth value corresponding to each effective pixel in the initial virtual left-eye scene image at the stereo depth distribution map based on the pixel content mapping relationship between the stereo depth distribution map and the initial virtual left-eye scene image.

[0116] Sub-step S252: For each first pixel to be repaired in the initial virtual left eye scene image, call the left eye repair template to determine multiple first valid reference pixels in the initial virtual left eye scene image that are located around the first pixel to be repaired.

[0117] In this embodiment, the first pixel to be repaired is a pixel belonging to a hole in the initial virtual left eye scene image that has not yet been repaired. The left eye repair template is used to describe how to select several effective pixels around the first pixel to be repaired to locate the effective pixels needed to repair the first pixel to be repaired, wherein the left eye repair template records the relative positional relationship between the corresponding first pixel to be repaired and the selected several effective pixels.

[0118] Sub-step S253: Based on the stereo depth value and pixel position of each effective pixel in the initial virtual left eye scene image, select multiple first calibration pixels in the initial virtual left eye scene image whose relative position distribution, depth level, and color features are consistent with multiple first effective reference pixels.

[0119] In this embodiment, if the absolute value of the difference between the stereo depth values ​​of two valid pixels is greater than a preset spacing threshold, it indicates that the two valid pixels are at different depth levels. After determining all the first valid reference pixels corresponding to a certain first pixel to be repaired, the control device 10 firstly determines, for each first valid reference pixel, multiple first valid pixels to be screened in the initial virtual left eye scene map that have the same depth level and color features as the first valid reference pixel. Then, it performs a comprehensive screening of the multiple first valid pixels to be screened corresponding to each of the first valid reference pixels according to the left eye repair template, to ensure that the multiple first valid pixels to be screened correspond to one first valid reference pixel, and that the relative position distribution between the multiple first valid pixels to be screened is consistent with the relative position distribution between all the first valid reference pixels. At this time, the multiple first valid pixels to be screened are the multiple first calibration pixels.

[0120] Sub-step S254: Call the left eye repair template to find the first effective substitute pixel in the initial virtual left eye scene map that is surrounded by multiple first calibration pixels, and use the first effective substitute pixel to perform pixel replacement processing on the first pixel to be repaired to obtain the effective pixel corresponding to the first pixel to be repaired.

[0121] In this embodiment, the relative positional relationship between the first effective substitute pixel and the plurality of first calibrated pixels is consistent with the relative positional relationship between the first pixel to be repaired and the corresponding plurality of first effective reference pixels. At this time, the foreground and background layers of the first effective substitute pixel and the first pixel to be repaired are basically consistent, and they also have basically similar textures.

[0122] Sub-step S255: The initial virtual left eye scene image after completing the pixel replacement operation of all the first pixels to be repaired is used as the target virtual left eye scene image.

[0123] In this embodiment, after performing the above pixel replacement operation on all the first pixels to be repaired in the initial virtual left eye scene image, it can be ensured that the texture effect of all the first pixels to be repaired is basically consistent after repair, and there will be no situation where the repaired foreground texture is mixed with foreground and background content, and there will be no blurring of the foreground edge.

[0124] Therefore, by executing the above sub-steps S251 to S255, this application can distinguish the foreground and background conditions during the repair process of the initial virtual left eye scene map, and avoid the phenomenon of blurred foreground edges due to the mixing of foreground and background content in the repaired foreground texture.

[0125] Alternatively, please refer to Figure 8 , Figure 8 yes Figure 2 The flowchart of step S250 includes the second sub-step. In this embodiment, the target virtual binocular scene map may include a target virtual right eye scene map. The step of performing image restoration on the initial virtual right eye scene map according to the stereo depth distribution map in step S250 to obtain the target virtual right eye scene map may include sub-steps S256 to S2510, so as to distinguish the foreground and background conditions during the restoration process of the initial virtual right eye scene map and avoid the phenomenon of blurred foreground edges due to the mixing of foreground and background content in the repaired foreground texture.

[0126] Sub-step S256: Determine the stereo depth value of each effective pixel in the initial virtual right eye scene map based on the stereo depth distribution map.

[0127] In this embodiment, the control device 10 can determine the stereo depth value corresponding to each effective pixel in the initial virtual right eye scene map at the stereo depth distribution map based on the pixel content mapping relationship between the stereo depth distribution map and the initial virtual right eye scene map.

[0128] Sub-step S257: For each second pixel to be repaired in the initial virtual right eye scene map, call the right eye repair template to determine multiple second valid reference pixels in the initial virtual right eye scene map that are located around the second pixel to be repaired.

[0129] In this embodiment, the second pixel to be repaired is a pixel belonging to a hole that has not yet been repaired in the initial virtual right eye scene image. The right eye repair template is used to describe how to select several effective pixels around the second pixel to be repaired to locate the effective pixels required to repair the second pixel to be repaired, wherein the right eye repair template records the relative positional relationship between the corresponding second pixel to be repaired and the selected several effective pixels.

[0130] Sub-step S258: Based on the stereo depth value and pixel position of each effective pixel in the initial virtual right eye scene image, select multiple second calibration pixels in the initial virtual right eye scene image whose relative position distribution, depth level, and color features are consistent with multiple second effective reference pixels.

[0131] In this embodiment, after determining all the second valid reference pixels corresponding to a certain second pixel to be repaired, the control device 10 first determines, for each second valid reference pixel, multiple second valid pixels to be screened in the initial virtual right eye scene map that are consistent with the second valid reference pixel in terms of depth level and color features. Then, it performs a comprehensive screening of the multiple second valid pixels to be screened corresponding to each of the second valid reference pixels according to the right eye repair template, so as to ensure that the multiple second valid pixels to be screened correspond to one second valid reference pixel, and the relative position distribution between the multiple second valid pixels to be screened is consistent with the relative position distribution between all the second valid reference pixels. At this time, the multiple second valid pixels to be screened are the multiple second calibration pixels.

[0132] Sub-step S259: Call the right eye repair template to find the second effective substitute pixel within the surrounding area of ​​multiple second calibration pixels in the initial virtual right eye scene image, and use the second effective substitute pixel to perform pixel replacement processing on the second pixel to be repaired to obtain the effective pixel corresponding to the second pixel to be repaired.

[0133] In this embodiment, the relative positional relationship between the second effective substitute pixel and the plurality of second calibrated pixels is consistent with the relative positional relationship between the second pixel to be repaired and the corresponding plurality of second effective reference pixels. At this time, the foreground and background layers of the second effective substitute pixel and the second pixel to be repaired are basically consistent, and they also have basically similar textures.

[0134] Sub-step S2510: The initial virtual right eye scene image after completing the pixel replacement operation of all second pixels to be repaired is used as the target virtual right eye scene image.

[0135] In this embodiment, after performing the above pixel replacement operation on all second pixels to be repaired in the initial virtual right eye scene image, it can be ensured that the texture effect of all second pixels to be repaired is basically consistent after repair, and there will be no situation where the repaired foreground texture is mixed with foreground and background content, and there will be no blurring of the foreground edge.

[0136] Therefore, by executing the above sub-steps S256 to S2510, this application can distinguish the foreground and background conditions during the repair process of the initial virtual right eye scene map, and avoid the phenomenon of blurred foreground edges due to the mixing of foreground and background content in the repaired foreground texture.

[0137] Therefore, by executing the above steps S210 to S250, this application can quickly construct binocular scene images with good image texture consistency of the robot under different viewpoints without adjusting the actual camera position configuration of the robot, so as to quickly construct the desired binocular training image set and reduce the cost of constructing the training image set.

[0138] The following is based on Figure 9 The image restoration effect of the target scene "aloe vera plant by the wall" is shown as an example to briefly explain the above virtual perspective image construction method. When the control device 10 controls the target robot to obtain the real binocular scene image of the scene "aloe vera plant by the wall" according to the current real perspective (wherein, the corresponding real left eye scene image is the real left eye scene image), the virtual perspective image construction method is briefly explained. Figure 9 (The background of the dotted frame shown is not a dark gray image). The control device 10 will determine the three-dimensional depth distribution map between the target robot and the scene "aloe vera potted plant by the wall" according to the above steps S220 and sub-steps S221 to S224. Figure 9 The background of the dotted frame shown is a dark gray image. Then, the control device 10 will construct an initial virtual left-eye scene map under the target virtual perspective based on the stereoscopic depth distribution map of the scene "aloe vera potted plant by the wall" and the real left-eye scene map, based on the parallax depth correlation relationship. Figure 9 The image shown has a black block in the lower right corner and is centered. The initial virtual right eye scene image (e.g.) Figure 9 The image shown has a black block in the lower left corner and is centered. Then, the control device 10 completes the image restoration work of the initial virtual left eye scene image and the initial virtual right eye scene image under the target virtual view by executing the above steps S250 and sub-steps S251 to S2510, and obtains a target virtual binocular scene image with good image texture consistency representing the target virtual view.

[0139] In this application, to ensure that the control device 10 can execute the aforementioned virtual perspective image construction method through the virtual perspective image construction device 100, this application implements the aforementioned functions by dividing the virtual perspective image construction device 100 into functional modules. The specific composition of the virtual perspective image construction device 100 provided in this application will be described below.

[0140] Please refer to Figure 10 , Figure 10 This is a schematic diagram of the composition of the virtual perspective image construction device 100 provided in this application embodiment. In this application embodiment, the virtual perspective image construction device 100 may include a real scene map acquisition module 110, a stereo depth map determination module 120, a virtual disparity map conversion module 130, a virtual scene map construction module 140, and a virtual scene map repair module 150.

[0141] The real scene image acquisition module 110 is used to acquire real binocular scene images of the target scene captured by the target robot from the current real perspective.

[0142] The stereo depth map determination module 120 is used to determine the stereo depth distribution map between the target robot and the target scene based on the real binocular scene map.

[0143] The virtual disparity map conversion module 130 is used to perform depth disparity conversion on the stereo depth distribution map based on the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual viewpoint, so as to obtain a virtual binocular disparity map of the target robot for the target scene under the target virtual viewpoint.

[0144] The virtual scene graph construction module 140 is used to perform image offset on the real binocular scene graph according to the virtual binocular disparity graph to obtain an initial virtual binocular scene graph that matches the target virtual viewpoint.

[0145] The virtual scene image repair module 150 is used to repair the initial virtual binocular scene image based on the stereo depth distribution map to obtain the target virtual binocular scene image.

[0146] It should be noted that the virtual perspective image construction apparatus 100 provided in this application embodiment has the same basic principle and technical effects as the aforementioned virtual perspective image construction method. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the above description of the virtual perspective image construction method.

[0147] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of the apparatus, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0148] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned readable storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0149] In summary, the virtual perspective image construction method, apparatus, control device, and readable storage medium provided in this application determine the stereo depth distribution map between the target robot and the target scene based on the real binocular scene map captured by the target robot from its current real perspective. Then, according to the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual perspective, the stereo depth distribution map is transformed by depth parallax to obtain a virtual binocular parallax map of the target robot for the target scene from the target virtual perspective. Next, image offset is performed based on the virtual binocular parallax map on the real binocular scene map to obtain an initial virtual binocular scene map that matches the target virtual perspective. Then, image inpainting is performed on the initial virtual binocular scene map based on the stereo depth distribution map to obtain a target virtual binocular scene map with good image texture consistency. Thus, without adjusting the robot's real camera position configuration, binocular scene images with good image texture consistency of the corresponding robot under different perspectives can be quickly constructed to quickly construct the desired binocular training image set and reduce the cost of constructing the training image set.

[0150] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for constructing virtual perspective images, characterized in that, The method includes: Acquire a real binocular scene image of the target robot from its current real-world perspective; A stereo depth distribution map between the target robot and the target scene is determined based on the real binocular scene map, wherein the stereo depth distribution map is used to characterize the distribution of spacing information between the target robot and various objects in the target scene; Based on the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual viewpoint, the stereo depth distribution map is subjected to depth parallax transformation to obtain a virtual binocular parallax map of the target robot for the target scene under the target virtual viewpoint. Based on the virtual binocular disparity map, the real binocular scene map is image offset to obtain an initial virtual binocular scene map that matches the target virtual viewpoint; Based on the stereo depth distribution map, the initial virtual binocular scene map is inpainted to obtain the target virtual binocular scene map; The real binocular scene map includes a real left-eye scene map and a real right-eye scene map. The step of determining the stereo depth distribution map between the target robot and the target scene based on the real binocular scene map includes: Feature extraction is performed on the real left-eye scene image and the real right-eye scene image respectively to obtain the corresponding left-eye scene feature image and right-eye scene feature image; Determine the disparity prediction distribution map between the left eye scene feature map and the right eye scene feature map for each of the pre-stored disparity values, wherein the disparity prediction distribution map corresponding to each disparity value is used to describe the distribution probability of the corresponding disparity value between the left eye scene feature map and the right eye scene feature map; The image is stitched together from the disparity prediction distribution maps corresponding to each disparity value, and the pre-stored disparity regression model is called to perform disparity regression on the stitched binocular disparity distribution map to obtain the real binocular disparity map corresponding to the current real viewpoint. Based on the current camera focal length of the target robot and the actual camera baseline distance corresponding to the current actual viewpoint, the actual binocular disparity map is subjected to disparity depth transformation to obtain the stereo depth distribution map.

2. The method according to claim 1, characterized in that, The step of determining the disparity prediction distribution map of each of the pre-stored multiple disparity values ​​between the left-eye scene feature map and the right-eye scene feature map includes: For each preset disparity value, the right eye scene feature map is shifted according to the disparity value in the feature extraction window of the right eye scene feature map to obtain the corresponding shifted feature map; The offset feature map corresponding to this disparity value is stitched together with the real left-eye scene image to obtain a binocular disparity feature map corresponding to this disparity value. The pre-stored disparity matching network model based on the Transformer model is invoked to perform pixel-level feature matching on the binocular disparity feature map corresponding to the disparity value, thereby obtaining the disparity prediction distribution map corresponding to the disparity value.

3. The method according to claim 1, characterized in that, The step of calling a pre-stored disparity regression model to perform disparity regression on the stitched binocular disparity distribution map to obtain a true binocular disparity map corresponding to the current true viewpoint includes: The binocular disparity distribution map is input into the disparity regression model for disparity distribution redistribution processing to obtain all possible disparity values ​​and their respective probabilities of occurrence at each pixel in the real binocular disparity map. For each pixel in the true binocular disparity map, a weighted summation is performed on all possible disparity values ​​corresponding to that pixel and the probability of occurrence of each of the possible disparity values ​​to obtain the true disparity value of the true binocular disparity map at that pixel.

4. The method according to any one of claims 1-3, characterized in that, The parallax-depth correlation required to perform a parallax-depth transformation operation or a depth-parallax transformation operation is expressed by the following formula: in, This value represents the parallax of the photographed object between the left-eye and right-eye cameras on the target robot, which are at the same baseline. This is used to represent the camera baseline distance between the left and right eye cameras of the target robot. The camera focal length used to represent the target robot. Used to represent the stereoscopic depth value between the photographed object and the target robot.

5. The method according to claim 1, characterized in that, The initial virtual binocular scene image includes an initial virtual left-eye scene image, and the target virtual binocular scene image includes a target virtual left-eye scene image. The step of performing image inpainting on the initial virtual left-eye scene image based on the stereo depth distribution map to obtain the target virtual left-eye scene image includes: The stereo depth value of each effective pixel in the initial virtual left eye scene image is determined based on the stereo depth distribution map. For each first pixel to be repaired in the initial virtual left eye scene image, the left eye repair template is called to determine multiple first valid reference pixels in the initial virtual left eye scene image that are located around the first pixel to be repaired. Based on the stereo depth value and pixel position of each effective pixel in the initial virtual left eye scene image, a plurality of first calibration pixels are selected in the initial virtual left eye scene image whose relative position distribution, depth level and color features are consistent with the plurality of first effective reference pixels. If the absolute value of the difference between the stereo depth values ​​of two effective pixels is greater than a preset spacing threshold, then the two effective pixels are at different depth levels. The left eye repair template is invoked to find the first effective substitute pixel within the encirclement of the plurality of first calibrated pixels in the initial virtual left eye scene image, and the first effective substitute pixel is used to perform pixel replacement processing on the first pixel to be repaired to obtain the effective pixel corresponding to the first pixel to be repaired. The initial virtual left eye scene image after completing the pixel replacement operation of all the first pixels to be repaired is used as the target virtual left eye scene image.

6. The method according to claim 5, characterized in that, The initial virtual binocular scene image further includes an initial virtual right eye scene image, and the target virtual binocular scene image further includes a target virtual right eye scene image. The step of performing image inpainting on the initial virtual right eye scene image based on the stereo depth distribution map to obtain the target virtual right eye scene image includes: The stereo depth value of each effective pixel in the initial virtual right eye scene image is determined based on the stereo depth distribution map. For each second pixel to be repaired in the initial virtual right eye scene image, the right eye repair template is called to determine multiple second valid reference pixels in the initial virtual right eye scene image that are located around the second pixel to be repaired. Based on the stereo depth value and pixel position of each effective pixel in the initial virtual right eye scene image, a plurality of second calibration pixels are selected in the initial virtual right eye scene image whose relative position distribution, depth level and color features are consistent with the plurality of second effective reference pixels. The right eye repair template is invoked to find the second effective substitute pixel within the encirclement of the plurality of second calibration pixels in the initial virtual right eye scene image, and the second effective substitute pixel is used to perform pixel replacement processing on the second pixel to be repaired to obtain the effective pixel corresponding to the second pixel to be repaired. The initial virtual right eye scene image after completing the pixel replacement operation for all second pixels to be repaired is used as the target virtual right eye scene image.

7. A virtual perspective image construction device, characterized in that, The device includes: The real scene image acquisition module is used to acquire the real binocular scene image captured by the target robot from the current real perspective of the target scene, wherein the real binocular scene image includes the real left eye scene image and the real right eye scene image. The stereo depth map determination module is used to determine the stereo depth distribution map between the target robot and the target scene based on the real binocular scene map, wherein the stereo depth map is used to characterize the distribution of distance information between the target robot and various objects in the target scene; The virtual disparity map conversion module is used to perform depth disparity conversion on the stereo depth distribution map based on the current camera focal length of the target robot and the virtual camera baseline distance corresponding to the target virtual viewpoint, so as to obtain a virtual binocular disparity map of the target robot for the target scene under the target virtual viewpoint. The virtual scene graph construction module is used to perform image offset on the real binocular scene graph based on the virtual binocular disparity graph to obtain an initial virtual binocular scene graph that matches the target virtual viewpoint; The virtual scene image restoration module is used to perform image restoration on the initial virtual binocular scene image based on the stereo depth distribution map to obtain the target virtual binocular scene image. The method by which the stereo depth map determination module determines the stereo depth distribution map between the target robot and the target scene based on the real binocular scene map includes: Feature extraction is performed on the real left-eye scene image and the real right-eye scene image respectively to obtain the corresponding left-eye scene feature image and right-eye scene feature image; Determine the disparity prediction distribution map between the left eye scene feature map and the right eye scene feature map for each of the pre-stored disparity values, wherein the disparity prediction distribution map corresponding to each disparity value is used to describe the distribution probability of the corresponding disparity value between the left eye scene feature map and the right eye scene feature map; The image is stitched together from the disparity prediction distribution maps corresponding to each disparity value, and the pre-stored disparity regression model is called to perform disparity regression on the stitched binocular disparity distribution map to obtain the real binocular disparity map corresponding to the current real viewpoint. Based on the current camera focal length of the target robot and the actual camera baseline distance corresponding to the current actual viewpoint, the actual binocular disparity map is subjected to disparity depth transformation to obtain the stereo depth distribution map.

8. A control device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor, the processor executing the computer program to implement the virtual perspective image construction method according to any one of claims 1-6.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the virtual perspective image construction method according to any one of claims 1-6.

Images