A method and system for perspective imaging of obstructions

Abstract

This invention belongs to the field of perspective imaging and provides a method and system for perspective imaging of occluded objects. The method includes acquiring a background image and an image of an occluded object under the same environment; dynamically identifying the occluded object in the image based on a pre-trained target detection model and obtaining an occluded object bounding box using rectangular boxes; processing the original occluded object image using the occluded object bounding box as a region of interest to obtain a processed image of interest; extracting the occluded object based on its HSV range according to the processed image of interest and performing binarization processing to obtain an occluded object image mask region; and superimposing the occluded object mask region with the mask region of the background image to obtain a visual perspective image of the occluded object.

Description

Technical Field

[0001] This invention belongs to the field of perspective imaging technology, specifically relating to a method and system for perspective imaging of obstructions. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] 3D camera-based perspective allows for the clear viewing of obstructions of a certain size, with relatively high precision and the ability to be applied to objects of various materials. This technology provides a foundation for improved visual manipulation of construction robots.

[0004] However, due to the complexity of certain installation environments, the diversity of construction processes, and the necessity of human-machine collaboration, existing industrial automation technologies cannot fully meet the operational requirements of construction robots. Traditional construction robots can only complete corresponding motion trajectories. However, they cannot see the environment behind obstructions, making installation work difficult. This results in problems such as low construction efficiency and difficulty in guaranteeing quality. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a method and system for perspective imaging of occluded objects. This invention can generate the effect of real-time tracking of occluded objects and making the area of ​​the occluded object to be processed transparent.

[0006] According to some embodiments, the first aspect of the present invention provides a method for perspective imaging of an obstruction, employing the following technical solution:

[0007] A method for perspective imaging of an obstruction includes:

[0008] Collect background images and images of obstructions under the same environment;

[0009] The occlusion objects in the occlusion image are dynamically identified based on the pre-trained object detection model, and the occlusion object annotation box is obtained by using rectangular box annotation.

[0010] The occluded object bounding box is used as the region of interest. The original occluded object image is then processed to obtain the image of interest to be processed.

[0011] Based on the image of interest to be processed, the occluder is extracted according to the HSV range of the occluder and binarized to obtain the mask region of the occluder image.

[0012] The masked area of ​​the occluded object is superimposed with the masked area of ​​the background image to obtain a visual perspective image of the occluded object.

[0013] Furthermore, the training process of the target detection model is as follows:

[0014] Obtain occlusion sample images in different poses to obtain a sample dataset;

[0015] The sample dataset is divided into a training set, a validation set, and a test set;

[0016] For each image in the sample dataset, label the occluder with a rectangular bounding box;

[0017] Normalize the length, width, and x and y coordinates of the center point of the bounding boxes obtained from all images in the training set, validation set, and test set to obtain the processed training set, validation set, and test set.

[0018] Based on the processed training set, the target detection model is trained to identify occlusions, and a preliminary trained target detection model is obtained.

[0019] Based on the validation set and the test set, the initially trained object detection model is validated and tested to obtain a well-trained object detection model.

[0020] Furthermore, based on the image of interest to be processed, the occluder is extracted according to its HSV range and binarized to obtain the occluder image mask region, specifically:

[0021] Based on the image of interest to be processed, the H, S, and V values ​​within the bounding box of the occluded object are statistically analyzed to obtain the maximum and minimum ranges of the HSV values ​​of the occluded object.

[0022] The occluded region is extracted using the minimum and maximum ranges of the three HSV values ​​of the occluded object, and then binarized.

[0023] The binarized image is used as the mask region for the occlusion image.

[0024] Furthermore, the occlusion region is extracted using the minimum and maximum ranges of the three HSV values ​​of the occlusion, and then binarized. Specifically:

[0025] Based on the image of interest to be processed, all pixels of the occluded object are extracted by inputting the minimum and maximum ranges of the three HSV values ​​of the occluded object as the upper and lower bounds of the HSV.

[0026] Using binarization, the extracted occluded pixels are assigned pure white, and the remaining pixels are assigned pure black.

[0027] The image obtained from the pure white area in the binarized image is used as the mask area.

[0028] Furthermore, the step of overlaying the masked area of ​​the occluder with the masked area of ​​the background image to obtain a visual perspective image of the occluder specifically involves:

[0029] The transparency of the synthesized transparent image is adjusted by changing the weights of the masked areas of the occluding object and the masked areas of the background image;

[0030] The masked area of ​​the occluded object is superimposed with the masked area of ​​the background image to obtain a visual perspective image of the occluded object.

[0031] Furthermore, when the weight of the mask region of the background image is 1 and the weight of the mask region of the occluder is 0, the occluder is in a state of complete transparency.

[0032] Furthermore, the target detection model adopts the Yolov5 model.

[0033] According to some embodiments, a second aspect of the present invention provides an obstruction-view imaging system, which adopts the following technical solution:

[0034] An obstruction-view imaging system includes:

[0035] The image acquisition module is configured to acquire background images and images of occluders in the same environment.

[0036] The occlusion recognition module is configured to dynamically identify occlusions in the occlusion image based on a pre-trained target detection model, and obtain occlusion annotation boxes using rectangular boxes.

[0037] The occlusion processing module is configured to use the occlusion bounding box as the region of interest to process the original occlusion image and obtain the image of interest to be processed.

[0038] The occlusion mask extraction module is configured to extract the occlusion based on the HSV range of the occlusion and perform binarization processing to obtain the occlusion image mask region based on the image of interest to be processed.

[0039] The perspective imaging module is configured to overlay the masked area of ​​the occluded object with the masked area of ​​the background image to obtain a visual perspective image of the occluded object.

[0040] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0041] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a perspective imaging method for an obstruction as described in the first aspect above.

[0042] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0043] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of a method for perspective imaging of an obstruction as described in the first aspect above.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] This invention innovates on interference prevention in traditional industrial robot vision processing. Traditional vision processing often suffers from reduced accuracy and correctness in identifying occluded objects due to the presence of interfering factors, such as people or other unprocessed occluders within the camera's field of view. This invention, however, uses a pre-trained target detection model to dynamically identify occluded objects in the image, significantly reducing the interference caused by these interfering factors.

[0046] This invention innovates on feedback mechanisms in traditional industrial robot operations. Existing industrial robot operations are generally based on human-robot collaboration, but due to obstructions affecting the operator's field of vision, the operator cannot obtain real-time positional relationships between obstructing and occluded objects, thus impacting the accuracy and speed of the operation. This invention, through dynamic recognition of obstructing objects in images and extraction of masked regions, provides real-time feedback to the operator on the positions of obstructing and occluded objects. This allows the operator to receive real-time feedback in various installation environments, resulting in higher installation efficiency and simpler, more convenient operation.

[0047] This invention proposes a method for perspective processing of obstructions in an installation environment, which can better assist robots in completing tasks in a human-robot collaborative work environment. Traditional industrial robots can only identify the relationship between obstructions and occluded objects through human visual perception during installation. This invention provides a perspective method and system based on a 3D camera, which can process images through a 3D camera to see the positions of obstructions and occluded objects in real time, automating and intelligentizing the installation process and greatly improving installation efficiency. The perspective method and system based on a 3D camera uses a transfer learning Yolov5 model algorithm. Compared with cascade training, it more accurately locates the area to be processed for obstructions and is not interfered with by other similar obstructions or moving objects in the external environment. It has advantages such as high accuracy, fast response, clean noise removal, and high recognition reliability. Attached Figure Description

[0048] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0049] Figure 1This is a flowchart of a perspective imaging method for obstructions according to an embodiment of the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0051] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0052] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0053] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0054] Example 1

[0055] like Figure 1 As shown, this embodiment provides a method for perspective imaging of obstructions. This embodiment uses the application of this method to a server as an example for illustration. It is understood that this method can also be applied to a terminal, or to a system including a terminal, a server, and a system, and is implemented through interaction between the terminal and the server. The server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In this embodiment, the method includes the following steps:

[0056] Step 1: Acquire background images and images of occluders in a specific environment;

[0057] Step 2: Dynamically identify occluders in the occluder image based on the pre-trained object detection model, and obtain the occluder annotation box using rectangular box annotation;

[0058] Step 3: Using the bounding boxes of the occluded objects as the region of interest, process the original image of the occluded objects to obtain the image of interest to be processed;

[0059] Step 4: Based on the image of interest to be processed, extract the occluder according to the HSV range of the occluder, and assign all pixels of the occluder to pure white and all other irrelevant pixels to pure black to obtain the mask area of ​​the occluder image.

[0060] Step 5: Overlay the masked area of ​​the occluded object with the masked area of ​​the background image to obtain a visual perspective image of the occluded object.

[0061] This embodiment discloses a perspective method and system based on a 3D camera. The 3D camera is equipped with a color camera light source. The perspective method and system include: acquiring a color image of an environment with occlusions; processing the color image and using a Yolov5 transfer learning model to obtain the corresponding Region of Interest (ROI) of the occlusions, where the ROI refers to the region of interest; extracting the closed area of ​​the occlusions within the ROIs using specific HSV values ​​of the occlusions, specifically noting that the HSV values ​​of the closed areas of the occlusions need to be approximately the same, where HSV refers to Hue (H), Saturation (S), and Value (V). The extracted occlusion pixels are assigned a value of 255 (pure white), and the remaining pixels are assigned a value of 0 (pure black). By performing a masking operation on this image and the background image, the occlusion area can be made transparent while the rest of the image remains transparent. The degree of transparency of the occlusion area can be controlled by superimposing different weights of the images.

[0062] In a specific embodiment, the method specifically includes:

[0063] Step S1: Use a 3D camera to capture a background image and an image with occlusions added while the background remains unchanged. The camera only needs to capture one frame of the background image; after adding occlusions, the camera captures images in real time. Natural factors such as lighting and weather in the background do not affect the image, but the background objects must remain unchanged. The occlusions here refer to movable objects of a certain size that can be captured by the camera.

[0064] Specifically, it includes:

[0065] S1.1: Use the Azure Kinect DK camera to collect color images of the installation environment;

[0066] (1) Industrial Camera

[0067] Product Name resolution Aspect Ratio FPS Rated FOV Color camera 1920*1080 16：9 0、5、15、30 90°*59°

[0068] (2) High-definition lens

[0069] Product Name Camera color space Formatting options High-definition lens BT.601 full range MJPEG chroma subsampling 4:2:2

[0070] S1.2: First, in a certain installation environment, remove the obstruction from the camera's shooting area so that the camera can capture one frame of the background image as the background.

[0071] S1.3: Then move into the area where the obstruction is located so that the camera can capture images of the object in the same installation environment.

[0072] Step S2: Train your own computer using the object detection model - YOLOv5 model, enabling it to dynamically identify occluded objects in images and mark the processing area with rectangular boxes in real time. Specifically, this includes:

[0073] S2.1: Use a camera to collect 3000 color images that you want to use to train the computer to recognize the image;

[0074] S2.2: Mark the occlusion processing areas in 3000 images with rectangles, and obtain the length, width, and x and y coordinates of the center point of the rectangles;

[0075] S2.3: Randomly shuffle the 3000 images to prevent overfitting during training;

[0076] S2.4: Images are divided into training set, validation set, and test set;

[0077] S2.5: Normalize the length, width, and x and y coordinates of the center point of the rectangles obtained from all images in the training set, validation set, and test set, and save the corresponding data into a text file as label data according to the format.

[0078] S2.6: Select version s, customize the dataset, and import the data into the Yolov5 model for training to obtain preliminary model files that have not been validated on the validation set and test set;

[0079] S2.7: Test and validate the model on an unlabeled dataset. Display the data in a window using the resulting bounding boxes. If a high accuracy is obtained, the final ideal model output is obtained.

[0080] S2.8; Encapsulate the class object, call the above model, and dynamically identify the occlusion processing area using the output model in the application scenario to obtain the corresponding rectangle.

[0081] Step S3: Using the aforementioned rectangle as the ROI region, process the original image. ROI refers to the region of interest or the region to be processed, resulting in the ROI image of the part to be processed. Specifically, this includes:

[0082] S3.1: Define a window with the same size and number of channels as the ROI region, and use the ROI algorithm to convert the original image into the image of the region to be processed and place it in this window.

[0083] Step S4: Find the average HSV value of the occluder in the area to be processed to obtain the HSV range of the occluder, thereby extracting the occluder. Assign all pixels of the occluder a value of 255 (pure white), and assign all other irrelevant pixels a value of 0 (pure black) to obtain the mask area, including:

[0084] S4.1: Select the callback function with the mouse, import the real-time obtained color image, enlarge it by 2 times and convert it from a color image to an HSV image, click the mouse to get the H, S, V values ​​returned for a specific occluded area and perform statistics.

[0085] S4.2: Use the algorithm to determine the maximum and minimum ranges of the three HSV values;

[0086] S4.3: Store the maximum and minimum values ​​in variables using a comparison algorithm;

[0087] S4.4: By using the upper and lower bounds of the three HSV values, the region of all pixels of the occluder (excluding external environment pixels) can be obtained and then binarized.

[0088] S4.5: After binarization, the extracted occluder pixels are assigned a value of 255 (pure white), and the remaining pixels are assigned a value of 0 (pure black);

[0089] S4.6: Use the image obtained from the white area as a mask image.

[0090] HSV refers to Hue (H), Saturation (S), and Value (V).

[0091] Step S5: Overlay the masked area of ​​the background image with the masked area of ​​the occluder to obtain a visual perspective of the occluder. The transparency can be changed, including:

[0092] S5.1: Overlay the masked area of ​​the background with the masked area of ​​the occluder. The transparency of the synthesized transparent image can be changed by changing the weights of the two images in real time.

[0093] S5.2: When the weight of the background mask area is 1 and the weight of the mask area containing the occluder is 0, the occluder is in a fully transparent state.

[0094] Perspective effect allows the background to be seen through the occluded object; therefore, perspective can be achieved through masking. By obtaining the pixel coordinates of the mask image obtained in step 4, and copying all pixels from the corresponding pixel coordinates in the background image to the corresponding pixel coordinates in the occluded object image, complete perspective of the occluded object can be achieved.

[0095] However, the perspective described above is complete perspective, and the transparency needs to be changed to facilitate visualization for staff. By overlaying the masked areas of the background image and the masked areas of the occluding object image, the transparency of the synthesized transparent image can be changed by altering the weights of the two images.

[0096] Example 2

[0097] This embodiment provides a perspective imaging system for obstructions, including:

[0098] The image acquisition module is configured to acquire background images and images of occluders in the same environment.

[0099] The occlusion recognition module is configured to dynamically identify occlusions in the occlusion image based on a pre-trained target detection model, and obtain occlusion annotation boxes using rectangular boxes.

[0100] The occlusion processing module is configured to use the occlusion bounding box as the region of interest to process the original occlusion image and obtain the image of interest to be processed.

[0101] The occlusion mask extraction module is configured to extract the occlusion based on the HSV range of the occlusion and perform binarization processing to obtain the occlusion image mask region based on the image of interest to be processed.

[0102] The perspective imaging module is configured to overlay the masked area of ​​the occluded object with the masked area of ​​the background image to obtain a visual perspective image of the occluded object.

[0103] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0104] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0105] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0106] Example 3

[0107] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the occlusion perspective imaging method described in Embodiment 1 above.

[0108] Example 4

[0109] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the occlusion perspective imaging method described in Embodiment 1 above.

[0110] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0111] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0112] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0113] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0114] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0115] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method of occlusion perspective imaging, characterized by, The method comprises the steps of: collecting background images and occlusion images in the same environment; based on a pre-trained target detection model, dynamically identifying the occlusion in the occlusion image, and obtaining an occlusion label frame by using a rectangular frame label; processing the original occlusion image by taking the occlusion label frame as a region of interest to obtain a processed image of interest; based on the processed image of interest, extracting the occlusion and performing binary processing according to the HSV range of the occlusion to obtain an occlusion image mask region, specifically comprising: based on the processed image of interest, counting the H, S and V values in the occlusion label frame to obtain the maximum and minimum ranges of the three values of the occlusion HSV; using the minimum and maximum ranges of the three values of the occlusion HSV to extract the occlusion region and perform binary processing thereon; the binary-processed image is taken as the occlusion image mask region; superimposing the occlusion mask region and the mask region of the background image to obtain a visual occlusion perspective image.

2. A method of occlusion perspective imaging according to claim 1, wherein, The training process of the target detection model comprises the steps of: obtaining occlusion sample images in different postures to obtain a sample data set; dividing the sample data set into a training set, a verification set and a test set; labeling the occlusion in each image in the sample data set with a rectangular label frame; normalizing the length, width, center point x and y coordinates of the rectangular frame obtained from all images in the training set, the verification set and the test set to obtain processed training set, verification set and test set; based on the processed training set, training the target detection model to identify the occlusion to obtain a preliminarily trained target detection model; based on the verification set and the test set, verifying and testing the preliminarily trained target detection model to obtain a trained target detection model.

3. A method of occlusion perspective imaging according to claim 1, wherein, The method of using the minimum and maximum ranges of the three values of the occlusion HSV to extract the occlusion region and perform binary processing thereon comprises the steps of: based on the processed image of interest, inputting the upper and lower bounds of HSV by using the minimum and maximum ranges of the three values of the occlusion HSV to extract all pixel points of the occlusion; using binary processing to assign the extracted occlusion pixel points as pure white and the remaining pixel points as pure black; the image obtained by taking the pure white region in the binary-processed image as a mask region.

4. The method of claim 1, wherein the step of generating a perspective view of the occluder comprises the step of: The method of superimposing the occlusion mask region and the mask region of the background image to obtain a visual occlusion perspective image comprises the steps of: ​ adjusting the transparency of the synthesized transparent image by changing the weight of the occlusion mask region and the mask region of the background image; superimposing the occlusion mask region and the mask region of the background image to obtain a visual occlusion perspective image.

5. A method of occlusion perspective imaging according to claim 4, wherein, When the weight of the mask region of the background image is 1 and the weight of the occlusion mask region is 0, the occlusion is in a completely transparent state.

6. The method of claim 1, wherein the step of generating a perspective view of the occluder comprises the step of: The target detection model adopts a Yolov5 model. ​ 7. A system for occlusion perspective imaging, characterized by The method comprises the steps of: an image acquisition module configured to collect background images and occlusion images in the same environment; an occlusion identification module configured to dynamically identify the occlusion in the occlusion image based on a pre-trained target detection model, and obtain an occlusion label frame by using a rectangular frame label; The occlusion processing module is configured to process the original occlusion image to obtain a to-be-processed image of interest by taking the occlusion bounding box as a region of interest. The occlusion mask extraction module is configured to extract the occlusion and perform binaryzation processing based on the to-be-processed image of interest according to the HSV range of the occlusion to obtain an occlusion image mask region, specifically as follows: Based on the to-be-processed image of interest, the H, S, and V values in the occlusion bounding box are counted to obtain the maximum range and the minimum range of the three values of the occlusion HSV; The occlusion region is extracted by using the minimum range and the maximum range of the three values of the occlusion HSV, and binaryzation processing is performed on the occlusion region; The binaryzation-processed image is taken as the occlusion image mask region; The perspective imaging module is configured to perform image superposition on the occlusion mask region and a mask region of a background image to obtain a visual occlusion perspective image.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the occlusion perspective imaging method according to any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the occlusion perspective imaging method according to any one of claims 1-6.

Images