A garbage rapid identification method based on an unmanned sanitation vehicle-mounted surround view camera

Abstract

The application discloses a garbage rapid identification method based on an all-around-view camera of an unmanned sanitation vehicle, and belongs to the technical field of image recognition, which comprises the following steps: acquiring original images with time stamp synchronization according to a multi-path unmanned vehicle-mounted fisheye camera, and obtaining bird's-eye view images through OpenGL rapid rendering by using a pixel-level fusion technology of internal and external parameter calibration; and performing garbage instance segmentation by using a Lang-Segment-Anything deep model to identify garbage coordinates. The bird's-eye view images are generated through all-around-view video splicing technology, and the Lang-Segment-Anything deep model trained is used to detect garbage on the bird's-eye view images, so that the application has lower memory occupation and hardware computing resource requirement, reduces the dependence of garbage rapid identification on training data, and solves the problems of single camera visual field limitation, target object crossing cameras and occlusion.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a method for quickly identifying garbage based on a surround-view camera carried by an unmanned sanitation vehicle. Background Art

[0002] Unmanned sanitation vehicles are an essential component of smart city development. Operating on open roads, they require highly dynamic environmental perception. Garbage inspection, a core function of unmanned sanitation vehicles, necessitates the use of garbage recognition technology, which plays a crucial role in improving operational quality and reducing energy consumption and emissions.

[0003] Currently, garbage image recognition technology deployed in unmanned sanitation vehicles is primarily based on monocular camera images. For mobile devices, single-camera object detection suffers from issues such as a limited field of view, objects crossing the camera, and occlusion, making it difficult to integrate with map information for better scene understanding. Currently, unmanned sanitation vehicles are generally equipped with multi-directional fisheye cameras. Bird's-eye-view object recognition algorithms (including garbage objects) based on multi-camera fusion can achieve better recognition results. However, these algorithms are often trained based on deep learning models and have high requirements for computing resources such as graphics cards and CPUs. Due to the limited computing resources in the vehicle environment, garbage detection algorithms based on a bird's-eye view are difficult to implement directly in industrial scenarios. The development of a more lightweight, generalizable, and real-time garbage recognition algorithm for industrial scenarios is a pressing issue. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention provides a method for rapid garbage identification based on a surround-view camera mounted on an unmanned sanitation vehicle. By using surround-view video stitching technology and a network model, garbage identification is performed in bird's-eye-view images, solving the problems of high computational requirements and high dependence on training data for garbage identification.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for quickly identifying garbage based on a surround-view camera on an unmanned sanitation vehicle, comprising the following steps:

[0006] S1: Acquires raw images with synchronized timestamps from multiple unmanned vehicle-mounted fisheye cameras;

[0007] S2: Processing the original image synchronized with the timestamp using the calibrated internal and external parameters to obtain an internal parameter mapping table and an external parameter mapping table;

[0008] S3: Based on the original images synchronized with the timestamp, the overlapping areas are manually selected and fused at the pixel level to generate a mask of the fused area weights;

[0009] S4: Based on the internal parameter mapping table, the external parameter mapping table and the mask of the fusion area weight, OpenGL is used for fast rendering to obtain a bird's-eye view image;

[0010] S5: Based on the bird's-eye view image, use the Lang-Segment-Anything deep model to perform garbage instance segmentation and obtain the garbage segmentation mask;

[0011] S6: According to the segmentation mask of the garbage, the garbage coordinates in the bird's-eye view image are obtained, and the garbage recognition coordinates in the vehicle body coordinate system are converted according to the calibrated internal and external parameters.

[0012] The beneficial effects of the present invention are as follows: the present invention generates a bird's-eye view image through surround video stitching technology, which solves the problem of single camera field of view limitation, target objects crossing the camera and occlusion, improves the performance of the garbage target recognition algorithm on unmanned sanitation vehicles, and is suitable for subsequent matching with map information;

[0013] The present invention uses a trained Lang-Segment-Anything depth model to perform garbage detection on bird's-eye view images, which has lower video memory usage and is based on OpenGL technology to achieve rapid generation of bird's-eye view images, meeting real-time requirements.

[0014] By using a trained Lang-Segment-Anything deep model, the present invention eliminates the need for building relevant data sets and subsequent model training, thereby reducing the dependence of garbage rapid identification algorithm model training on data.

[0015] Furthermore, the specific steps of S2 are as follows:

[0016] S201: Based on the calibrated internal parameters, the image distortion correction function InitUndistortRectifyMap is used to perform processing to obtain an internal parameter mapping table;

[0017] S202: Mapping pixels of the original image synchronized with the timestamp according to the internal parameter mapping table to obtain a corrected image;

[0018] S203: performing region cropping based on the corrected image to obtain a cropped image;

[0019] S204: Projecting the cropped image to a vehicle body bird's-eye view coordinate system according to the calibrated external parameters to obtain an external parameter mapping table.

[0020] The beneficial effect of the above further solution is: the original image synchronously collected by the fisheye camera is processed by internal parameters and external parameters to obtain an external parameter mapping table and an internal parameter mapping table, which facilitates the subsequent generation of a bird's-eye view image.

[0021] Furthermore, the specific steps of S3 are as follows:

[0022] S301: manually selecting overlapping areas based on the original images synchronized with the timestamps to obtain overlapping areas between the original images;

[0023] S302: defining a fusion basic variable and a fusion weight function according to the original image synchronized with the timestamp and the selected overlapping area;

[0024] S303: Based on the basic fusion variables and the fusion weight function, the distance of each point relative to the boundary of the fusion area is calculated pixel by pixel, and the fusion weight of each point for the two overlapping images is generated to obtain a mask of the fusion area weight.

[0025] The beneficial effect of the above further solution is that by determining the mask of the fusion area weight, data can be provided for the generation of the bird's-eye view image, thereby reducing the memory usage.

[0026] Furthermore, the specific steps of S4 are as follows:

[0027] S401: Normalizing the bird's-eye view image coordinate system to obtain normalized coordinate points;

[0028] S402: The normalized coordinate point is used as vertex data and input into the OpenGL vertex shader for processing to obtain vertex coordinates and input into the OpenGL fragment shader;

[0029] S403: performing an inverse top-down transformation on the vertex coordinates using an OpenGL fragment shader according to the external parameter mapping table, and performing an inverse correction transformation on the vertex coordinates according to the internal parameter mapping table to obtain texture coordinates;

[0030] S404: performing weighted rendering of the fusion area on the texture coordinates using an OpenGL fragment shader according to the mask of the fusion area weight to obtain rendering coordinates;

[0031] S405: Mapping is performed using the vertex coordinates, and rendering is performed using the rendering coordinates to obtain a bird's-eye view image.

[0032] The beneficial effects of the above further solution are: generating a bird's-eye view image through OpenGL rendering has lower video memory occupancy when performing garbage detection, and OpenGL can realize the rapid generation of bird's-eye view images, meeting the real-time requirements of rapid garbage identification.

[0033] Furthermore, the specific implementation of S5 is as follows:

[0034] Input garbage-related prompt words into the Lang-Segment-Anything deep model, and use the Lang-Segment-Anything deep model to segment garbage instances in the bird's-eye view image to obtain a garbage segmentation mask; the Lang-Segment-Anything deep model includes: a Segment-Anything image segmentation model and a GroundingDINO object detection model.

[0035] The beneficial effects of the above further scheme are: garbage identification is performed through the Lang-Segment-Anything deep segmentation model, and the public model can be directly used without the need for model construction and training, which reduces the dependence of garbage identification on training data and reduces the demand for computing resources of the garbage rapid identification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for rapid garbage identification based on a surround-view camera mounted on an unmanned sanitation vehicle;

[0037] Figure 2 It is a diagram of the fusion basic variable form;

[0038] Figure 3 Rendering flow graph for OpenGL. DETAILED DESCRIPTION

[0039] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0040] like Figure 1 As shown in the figure, a flow chart of a method for rapid garbage identification based on a surround-view camera mounted on an unmanned sanitation vehicle includes the following steps:

[0041] S1: Acquires raw images with synchronized timestamps from multiple unmanned vehicle-mounted fisheye cameras;

[0042] S2: Processing the original image synchronized with the timestamp using the calibrated internal and external parameters to obtain an internal parameter mapping table and an external parameter mapping table; wherein the calibrated internal and external parameters are the calibrated internal parameters and external parameters of the unmanned vehicle-mounted fisheye camera;

[0043] S3: Based on the original images synchronized with the timestamp, the overlapping areas are manually selected and fused at the pixel level to generate a mask of the fused area weights;

[0044] S4: Based on the internal parameter mapping table, the external parameter mapping table and the mask of the fusion area weight, OpenGL is used for fast rendering to obtain a bird's-eye view image;

[0045] S5: Based on the bird's-eye view image, use the Lang-Segment-Anything deep model to perform garbage instance segmentation and obtain the garbage segmentation mask;

[0046] S6: According to the segmentation mask of the garbage, the garbage coordinates in the bird's-eye view image are obtained, and the garbage recognition coordinates in the vehicle body coordinate system are converted according to the calibrated internal and external parameters.

[0047] In one embodiment of the present invention, the specific steps of S2 are as follows:

[0048] S201: Based on the calibrated internal parameters, the image distortion correction function InitUndistortRectifyMap is used to perform processing to obtain a mapping table from original camera pixels to corrected image pixels, i.e., an internal parameter mapping table;

[0049] S202: Mapping pixels of the original image synchronized with the timestamp according to the internal parameter mapping table to obtain a corrected image;

[0050] S203: performing region cropping based on the corrected image to remove unnecessary edge portions to obtain a cropped image;

[0051] S204: Based on the calibrated external parameters, the cropped image is projected onto the vehicle body bird's-eye view coordinate system to obtain a mapping table from the corrected image pixels to the vehicle body bird's-eye view image pixels, i.e., an external parameter mapping table. The vehicle body bird's-eye view coordinate system is defined according to usage requirements to determine the range and resolution of the bird's-eye view image display.

[0052] In one embodiment of the present invention, the specific steps of S3 are as follows:

[0053] S301: manually selecting overlapping areas based on the original images synchronized with the timestamps to obtain overlapping areas between the original images;

[0054] S302: Based on the original image synchronized with the timestamp and the selected overlapping area, define the basic variables of fusion and the fusion weight function; wherein, Figure 2 As shown in the figure, it is a fusion basic variable form diagram. The fusion basic variable is the distance between the pixel point and the boundary of the overlapping area, that is, Figure 2The dashed line segments D1 and D2 in the image, area1 and area2 are non-overlapping areas of the image, and multiple methods can be used to calculate the distance, such as Figure 2 The middle left picture shows the distance obtained by directly calculating the point perpendicular to the boundary line. Figure 2 The middle right image shows a straight line with a specified slope for the pixel point. The distance from the point to the intersection of the straight line and the boundary is used to obtain the expression of the fusion weight function as follows:

[0055] ,

[0056] in, is the fusion weight of the pixel on image i, is the transformation function, is the distance of the pixel relative to the boundary of the overlapping area of ​​image i, is the distance of the pixel relative to the boundary of the overlapping area of ​​image j.

[0057] S303: Based on the basic fusion variables and the fusion weight function, the distance of each point relative to the boundary of the fusion area is calculated pixel by pixel, and the fusion weight of each point for the two overlapping images is generated to obtain the mask of the fusion area weight. The mask of the fusion area weight is used to fuse the overlapping areas at the pixel level.

[0058] In one embodiment of the present invention, Figure 3 As shown in the figure, it is an OpenGL rendering flow chart. First, the left point of the bird's-eye view image is normalized to obtain the GL vertex data of the target image and input it into the OpenGL vertex shader for processing. Then, it is passed to the fragment shader for processing. The inverse top-down transformation is performed through the external parameter mapping table, and the inverse correction transformation is performed through the internal parameter mapping table to obtain the texture coordinate point. The texture coordinate point is fused weighted by the fusion mask. Finally, the output of the OpenGL vertex shader is mapped and rendered in combination with the output of the OpenGL fragment shader to obtain the screen coordinate. The present invention uses OpenGL to generate a bird's-eye view image. The specific steps of S4 are as follows:

[0059] S401: Normalizing the bird's-eye view image coordinate system to obtain normalized coordinate points;

[0060] S402: The normalized coordinate point is used as vertex data and input into the OpenGL vertex shader for processing to obtain vertex coordinates and input into the OpenGL fragment shader;

[0061] S403: performing an inverse top-down transformation on the vertex coordinates using an OpenGL fragment shader according to the external parameter mapping table, and performing an inverse correction transformation on the vertex coordinates according to the internal parameter mapping table to obtain texture coordinates;

[0062] S404: performing weighted rendering of the fusion area on the texture coordinates using an OpenGL fragment shader according to the mask of the fusion area weight to obtain rendering coordinates;

[0063] S405: Mapping is performed using vertex coordinates, and rendering is performed using rendering coordinates to obtain a bird's-eye view image. OpenGL is rendered based on a GPU, and a parallel rendering process is implemented using a shader, which speeds up the generation of the bird's-eye view image and can meet the real-time requirements of garbage detection and recognition.

[0064] In one embodiment of the present invention, the Lang-Segment-Anything deep model in S5 is an existing model, which can input garbage-related prompt words, such as "Rubbish", into the Lang-Segment-Anything deep model, so that the Lang-Segment-Anything deep model can generate a mask for the garbage in the EBV bird's-eye view image, perform garbage instance segmentation on the bird's-eye view image, and obtain a garbage segmentation mask; the Lang-Segment-Anything deep model includes: a Segment-Anything image segmentation model and a GroundingDINO target detection model. This embodiment can directly use the existing trained model without the need for additional data processing and training, thereby reducing the dependence of the garbage rapid identification algorithm model training on data and reducing the amount of data calculation.

[0065] In one embodiment of the present invention, the internal parameter mapping table, the external parameter mapping table and the mask of the fusion area weight in the present invention do not need to be updated after being generated once. When the present invention is actually running, it can meet the real-time requirements of bird's-eye view image generation by directly rendering through OpenGL, and the image is in the video memory and can be directly used for segmentation algorithm calculation.

[0066] The beneficial effects of the present invention are as follows: the present invention generates a bird's-eye view image through surround video stitching technology, which solves the problem of single camera field of view limitation, target objects crossing the camera and occlusion, improves the performance of the garbage target recognition algorithm on unmanned sanitation vehicles, and is suitable for subsequent matching with map information;

[0067] The present invention uses a trained Lang-Segment-Anything depth model to perform garbage detection on bird's-eye view images, which has lower video memory usage and is based on OpenGL technology to achieve rapid generation of bird's-eye view images, meeting real-time requirements.

[0068] The application reduces the dependence of the garbage rapid recognition algorithm model training on data by using the trained Lang-Segment-Anything deep model without constructing a related data set and subsequent model training.

Claims

1. A method for rapid garbage identification based on a surround-view camera mounted on an unmanned sanitation vehicle, characterized in that: The following steps are involved: S1: Acquires raw images with synchronized timestamps from multiple unmanned vehicle-mounted fisheye cameras; S2: Processing the original image synchronized with the timestamp using the calibrated internal and external parameters to obtain an internal parameter mapping table and an external parameter mapping table; S3: Based on the original images synchronized with the timestamp, the overlapping areas are manually selected and fused at the pixel level to generate a mask of the fused area weights; S4: Based on the internal parameter mapping table, the external parameter mapping table and the mask of the fusion area weight, OpenGL is used for fast rendering to obtain a bird's-eye view image; S5: Based on the bird's-eye view image, use the Lang-Segment-Anything deep model to perform garbage instance segmentation and obtain the garbage segmentation mask; S6: According to the segmentation mask of the garbage, the garbage coordinates in the bird's-eye view image are obtained, and the garbage recognition coordinates in the vehicle body coordinate system are converted according to the calibrated internal and external parameters.

2. The method for rapid garbage identification based on the surround-view camera on an unmanned sanitation vehicle according to claim 1 is characterized in that: The specific steps of S2 are as follows: S201: Based on the calibrated internal parameters, the image distortion correction function InitUndistortRectifyMap is used to perform processing to obtain an internal parameter mapping table; S202: Mapping pixels of the original image synchronized with the timestamp according to the internal parameter mapping table to obtain a corrected image; S203: performing region cropping based on the corrected image to obtain a cropped image; S204: Projecting the cropped image to a vehicle body bird's-eye view coordinate system according to the calibrated external parameters to obtain an external parameter mapping table.

3. The method for rapid garbage identification based on the surround-view camera on an unmanned sanitation vehicle according to claim 1 is characterized in that: The specific steps of S3 are as follows: S301: manually selecting overlapping areas based on the original images synchronized with the timestamps to obtain overlapping areas between the original images; S302: defining a fusion basic variable and a fusion weight function according to the original image synchronized with the timestamp and the selected overlapping area; S303: Based on the basic fusion variables and the fusion weight function, the distance of each point relative to the boundary of the fusion area is calculated pixel by pixel, and the fusion weight of each point for the two overlapping images is generated to obtain a mask of the fusion area weight.

4. The method for rapid garbage identification based on the surround-view camera on an unmanned sanitation vehicle according to claim 1 is characterized in that: The specific steps of S4 are as follows: S401: Normalizing the bird's-eye view image coordinate system to obtain normalized coordinate points; S402: The normalized coordinate point is used as vertex data and input into the OpenGL vertex shader for processing to obtain vertex coordinates and input into the OpenGL fragment shader; S403: performing an inverse top-down transformation on the vertex coordinates using an OpenGL fragment shader according to the external parameter mapping table, and performing an inverse correction transformation on the vertex coordinates according to the internal parameter mapping table to obtain texture coordinates; S404: performing weighted rendering of the fusion area on the texture coordinates using an OpenGL fragment shader according to the mask of the fusion area weight to obtain rendering coordinates; S405: Mapping is performed using the vertex coordinates, and rendering is performed using the rendering coordinates to obtain a bird's-eye view image.

5. The method for rapid garbage identification based on the surround-view camera on an unmanned sanitation vehicle according to claim 1 is characterized in that: The specific implementation of S5 is as follows: Input garbage-related prompt words into a trained Lang-Segment-Anything deep model, and use the trained Lang-Segment-Anything deep model to perform garbage instance segmentation on the bird's-eye view image to obtain a garbage segmentation mask; the trained Lang-Segment-Anything deep model is an existing model, including: a Segment-Anything image segmentation model and a GroundingDINO object detection model.

Images