Image detection method and device based on sparse point cloud, equipment and storage medium

Abstract

The invention provides an image detection method, device and equipment based on sparse point cloud and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning and the like, and can be applied to scenes of automatic driving, robot navigation and the like. The scheme comprises the steps of obtaining a to-be-processed image, and determining sparse feature data of the to-be-processed image; the to-be-processed image is sparse point cloud data, and the sparse feature data represents non-zero pixel points in the to-be-processed image; determining a spatial index and a sequence index of non-zero pixel points; the spatial index represents the coordinate position of the non-zero pixel point in the to-be-processed image, and the sequential index represents the pixel point sorting number of the non-zero pixel point in the to-be-processed image; and performing target detection on the to-be-processed image according to the sparse feature data and the spatial index and the sequence index of each non-zero pixel point. By determining the spatial index and the sequential index of the non-zero pixel points, the limitation on the shape of the convolution kernel is reduced, and the target detection precision is improved.

Description

Technical Field

[0001] The present disclosure relates to computer vision, deep learning and other technical fields in the field of artificial intelligence, and can be applied to scenes such as autonomous driving and robot navigation, and in particular to an image detection method, device, equipment and storage medium based on sparse point cloud. Background Art

[0002] With the rapid development of 3D technology, three-dimensional data has been widely used in the fields of autonomous driving, robot navigation, industrial inspection, etc. Three-dimensional data can be in the form of point cloud data, and the distribution of point clouds can represent the distribution of objects in the environment.

[0003] In sparse point cloud data, due to the scattered distribution of point clouds, traditional convolutional neural networks have problems such as large computational complexity, high memory usage, and low convolution kernel utilization when performing image processing, which affects the detection accuracy and efficiency of objects in the image. Summary of the invention

[0004] The present invention provides a sparse point cloud-based image detection method, device, equipment and storage medium.

[0005] According to a first aspect of the present disclosure, there is provided an image detection method based on a sparse point cloud, comprising:

[0006] Acquire an image to be processed, and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed;

[0007] Determine the spatial index and sequential index of the non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel point sorting number of the non-zero pixel point in the image to be processed;

[0008] According to the sparse feature data, and the spatial index and the sequential index of each non-zero pixel point, target detection is performed on the object in the image to be processed.

[0009] According to a second aspect of the present disclosure, there is provided an image detection device based on a sparse point cloud, comprising:

[0010] An image acquisition unit, used to acquire an image to be processed and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed;

[0011] An index determination unit, used to determine the spatial index and sequential index of the non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel point sorting number of the non-zero pixel point in the image to be processed;

[0012] The image detection unit is used to perform target detection on the object in the image to be processed according to the sparse feature data and the spatial index and sequential index of each non-zero pixel point.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively coupled to the at least one processor;

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the present disclosure when executed by a processor.

[0019] According to the technology disclosed in the present invention, the flexibility of convolution on sparse point cloud data is improved, thereby improving the accuracy and efficiency of target detection.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flow chart of an image detection method based on sparse point cloud provided according to an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of non-zero pixels in an image to be processed provided according to an embodiment of the present disclosure;

[0024] Figure 3 is a flow chart of an image detection method based on sparse point cloud provided according to an embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of spatial grouping of initial areas provided according to an embodiment of the present disclosure;

[0026] Figure 5a is a priority diagram of a nearest neighbor search provided according to an embodiment of the present disclosure;

[0027] Figure 5b is a priority diagram of the farthest neighbor search provided according to an embodiment of the present disclosure;

[0028] Figure 6 is a flow chart of an image detection method based on sparse point cloud provided according to an embodiment of the present disclosure;

[0029] Figure 7 is a schematic diagram of sequential indexing of non-zero pixels provided according to an embodiment of the present disclosure;

[0030] Figure 8 is a structural block diagram of an image detection device based on sparse point cloud provided according to an embodiment of the present disclosure;

[0031] Fig. 9 is a structural block diagram of an image detection device based on sparse point cloud provided according to an embodiment of the present disclosure;

[0032] Fig.10 is a block diagram of an electronic device for implementing the image detection method based on sparse point cloud according to an embodiment of the present disclosure;

[0033] Fig.11 It is a block diagram of an electronic device used to implement the image detection method based on sparse point cloud according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] With the rapid development of 3D acquisition technology, 3D data acquired by sensors such as LiDAR and RGB-D (Red Green Blue-Depth) cameras have been widely used in autonomous driving, robot navigation, industrial inspection and other fields. However, for sparse point cloud data, traditional convolutional neural networks have problems such as large amount of computation, high memory usage, and low convolution kernel utilization when processing sparse point cloud data. Especially in 3D object detection and segmentation tasks, how to improve the efficiency and accuracy of sparse convolution has become a research hotspot.

[0036] Submanifold sparse convolution can reduce unnecessary calculations by directly performing convolution operations on sparse point clouds, but the shape of the convolution kernel of submanifold sparse convolution is fixed, which is difficult to adapt to the structure of irregular sparse point clouds, limiting the ability of feature extraction. The current submanifold sparse convolution can increase the receptive field by increasing the size of the convolution kernel, but a larger receptive field will affect the accuracy and efficiency of target detection. Therefore, how to dynamically adjust the shape of the convolution kernel to better fit the structure of the sparse point cloud is the key to improving the performance of sparse convolution.

[0037] The present invention discloses a sparse point cloud-based image detection method, apparatus, device and storage medium, which are applied to computer vision, deep learning and other technical fields in the field of artificial intelligence, and can be applied to scenes such as autonomous driving and robot navigation to improve the flexibility of convolution, thereby improving the accuracy and efficiency of target detection.

[0038] It should be noted that the data in this embodiment is not data for a specific user and cannot reflect the personal information of a specific user. It should be noted that the images in this embodiment are from a public data set.

[0039] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0040] In order to enable readers to more deeply understand the implementation principle of the present disclosure, the following Figure 1-Figure 11 The embodiment is further refined.

[0041] Figure 1 FIG. 1 is a flow chart of a sparse point cloud-based image detection method according to an embodiment of the present disclosure, which can be performed by a sparse point cloud-based image detection device. Figure 1 As shown, the method comprises the following steps:

[0042] S101, obtaining an image to be processed, and determining sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixel points in the image to be processed.

[0043] Exemplarily, the image to be processed can be acquired by a laser radar, RGB-D or other sensors, and the image to be processed can be point cloud data. In this embodiment, the image to be processed is sparse point cloud data. The point cloud in the sparse point cloud data is sparsely distributed or irregularly distributed, with some positions being zero pixels and some positions being non-zero pixels, that is, the pixels in the image to be processed include zero pixels and non-zero pixels. A zero pixel means that there is no valid image pixel information at the pixel, and a non-zero pixel means that there is image pixel information at the location.

[0044] After obtaining the image to be processed, the image to be processed can be first convolved according to a preset convolution layer to obtain 2D sparse features of the image to be processed as sparse feature data. In this embodiment, the sparse feature data can represent non-zero pixels in the image to be processed, that is, the sparse feature data represents the non-zero features of the image to be processed. In this embodiment, the convolution process for obtaining 2D sparse features is not specifically limited.

[0045] The sparse feature data may be an N×C matrix vector, where N is the number of non-zero features, that is, the number of non-zero pixels, and C is the feature dimension, which may be the number of channels of the image feature, for example, the number of RGB channels. Figure 2 is a schematic diagram of non-zero pixels in the image to be processed. Figure 2 Indicated by black squares, Figure 2 There are 6 non-zero pixels in , that is, N is 6.

[0046] S102, determining the spatial index and sequential index of the non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel ordering number of the non-zero pixel point in the image to be processed.

[0047] Exemplarily, each pixel in the image corresponds to its own coordinate position, indicating the distribution of the pixel in the image. The coordinate position of the non-zero pixel in the image to be processed is determined as the spatial index of the non-zero pixel, that is, the spatial index represents the coordinate position of the non-zero pixel in the image to be processed.

[0048] For each non-zero pixel, determine the sequential index of the non-zero pixel. The sequential index represents the pixel ordering number of the non-zero pixel in the image to be processed, and the sequential index can be an integer starting from 0. That is, starting from 0, number each non-zero pixel. For example, there are 6 non-zero pixels in the image to be processed, and the non-zero pixels can be sorted starting from 0 according to the order of the non-zero pixels in the image to be processed. The sequential indexes of the non-zero pixels are 0, 1, 2, 3, 4, and 5 respectively. Starting from the first row of pixels, the non-zero pixels can be searched in the order of the pixels from left to right, and the non-zero pixels can be numbered.

[0049] S103 , performing target detection on the object in the image to be processed according to the sparse feature data and the spatial index and sequential index of each non-zero pixel point.

[0050] Exemplarily, there may be one or more objects in the image to be processed, and the point cloud distribution of the locations of the objects in the image to be processed may be relatively dense. For example, the image to be processed is a road image, and the objects in the image to be processed may be vehicles or pedestrians on the road.

[0051] The sparse feature data can be processed according to the spatial index and sequential index of all non-zero pixels in the image to be processed. For example, the relevant information of the object can be extracted from the image to be processed according to the spatial index and sequential index of each non-zero pixel, and the feature vector obtained after extraction is used as the target feature vector. That is, the target feature vector can characterize the object in the image to be processed. According to the target feature vector, the image to be processed is segmented to obtain the object in the image to be processed, that is, the target detection of the image to be processed is completed, and the detected target is the object in the image to be processed.

[0052] A convolution kernel of a preset size can be pre-set, and according to the convolution kernel, sparse feature data is processed based on the spatial index and sequential index of non-zero pixels. For example, the preset convolution kernel is a sparse convolution, specifically, a submanifold sparse convolution. The convolution kernel can be translated according to the spatial index of the non-zero pixel, and the convolution kernel can be moved to each non-zero pixel. For example, the center position of the convolution kernel can be moved to the coordinate position of the non-zero pixel. If the convolution kernel is 5×5 in size, the position of the third row and the third column in the convolution kernel can be placed at each non-zero pixel in sequence. For each convolution kernel at a non-zero pixel, the non-zero features of the non-zero pixel near the convolution kernel are convoluted according to the sequential index of each non-zero pixel. For example, the non-zero pixel located outside the convolution kernel and adjacent to the convolution kernel can be determined. When performing the convolution, the zero pixel in the convolution kernel is not processed, but the non-zero pixel next to it is processed to achieve the change of the shape and size of the convolution kernel. The processing results at each position are combined to obtain the target feature vector. For example, the processing results of the convolution kernel at each non-zero pixel point can be added to obtain the target feature vector.

[0053] In the disclosed embodiment, for an image to be processed of a sparse point cloud, non-zero features in the image to be processed are determined as sparse point cloud data. The spatial index and sequential index of each non-zero pixel in the image to be processed are determined, and the sparse feature data is processed according to the spatial index and sequential index of each non-zero pixel, so as to achieve accurate processing of non-zero pixels at different positions, avoid limitations on the size of the convolution shape during processing, thereby adapting to the structure of irregular point cloud data, and being able to quickly and accurately identify target objects from the image to be processed. By processing non-zero pixels, the amount of computation required for processing point cloud data is greatly reduced, the ability to extract features is improved, and the accuracy and efficiency of target detection are thereby improved.

[0054] Figure 3 A flowchart of an image detection method based on sparse point cloud provided in an embodiment of the present disclosure.

[0055] In this embodiment, target detection is performed on objects in the image to be processed based on sparse feature data, and the spatial index and sequential index of each non-zero pixel point, including: determining the neighboring area of ​​the non-zero pixel point in the image to be processed based on the spatial index of the non-zero pixel point; wherein the neighboring area represents a preset area range adjacent to the non-zero pixel point in the image to be processed; if it is determined that there is a target pixel point in the neighboring area of ​​the non-zero pixel point, then determining the sequential index of the target pixel point; wherein the target pixel point represents other non-zero pixels in the image to be processed; target detection is performed on objects in the image to be processed based on the sparse feature data, the sequential index of each non-zero pixel point, and the sequential index of the target pixel points corresponding to each non-zero pixel point.

[0056] This embodiment is based on the above embodiment. Figure 3 As shown, the method comprises the following steps:

[0057] S301, obtaining an image to be processed, and determining sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixel points in the image to be processed.

[0058] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0059] S302, determining the spatial index and sequential index of the non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel ordering number of the non-zero pixel point in the image to be processed.

[0060] Exemplarily, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0061] S303, determining a neighboring region of the non-zero pixel in the image to be processed according to a spatial index of the non-zero pixel; wherein the neighboring region represents a preset region range adjacent to the non-zero pixel in the image to be processed.

[0062] Exemplarily, the spatial index can be a representation of coordinates, for example, the spatial index is (x, y), which represents the position of the non-zero pixel in the image to be processed. For each non-zero pixel, the adjacent area near the position of the non-zero pixel is determined from the image to be processed according to the spatial index of the non-zero pixel. The adjacent area is the neighborhood of the non-zero pixel, that is, the preset area range adjacent to the non-zero pixel. For example, the area of ​​two pixels above the non-zero pixel can be used as the adjacent area of ​​the non-zero pixel.

[0063] A non-zero pixel may correspond to one or more neighboring regions. For example, a neighboring region may be determined above, below, left, and right of the non-zero pixel. Multiple different preset region ranges may be preset, that is, the sizes of multiple neighboring regions of a non-zero pixel may be different. For example, the neighboring region directly above the non-zero pixel is two pixels in size, and the neighboring region to the upper left of the non-zero pixel is four pixels in size.

[0064] In this embodiment, the neighboring area of ​​the non-zero pixel in the image to be processed is determined according to the spatial index of the non-zero pixel, including: determining an initial area of ​​a preset size according to the spatial index of the non-zero pixel; wherein the non-zero pixel is located in the initial area; and dividing the initial area into at least two neighboring areas of the non-zero pixel.

[0065] Specifically, the size of the initial region is preset, and the initial region of the preset size is obtained according to the spatial index of the non-zero pixel point and the preset size. For example, a 5×5 initial region is preset. Each non-zero pixel point corresponds to an initial region of a preset size, and the non-zero pixel point is located in the corresponding initial region. For example, if there are 6 non-zero pixels on the image to be processed, there are 6 initial regions, and there is an initial region of 5×5 size at the position of each non-zero pixel point, and the non-zero pixel point is a pixel point in 5×5.

[0066] The initial region is spatially grouped and divided into multiple parts. The divided parts can be used as neighboring regions of non-zero pixels, and multiple neighboring regions of non-zero pixels can be obtained. For example, the pixels other than non-zero pixels in the initial region can be grouped to obtain multiple neighboring regions, and one neighboring region can include one or more pixels.

[0067] The beneficial effect of this setting is that the initial area is spatially grouped, which makes it easier to find non-zero pixels from different spaces, and to find non-zero pixels in different spaces during the subsequent convolution processing of feature extraction, thereby changing the shape of the convolution kernel, improving the ability of feature extraction, and improving the efficiency and accuracy of target detection.

[0068] In this embodiment, the non-zero pixel point is located at the center of the initial area; the initial area is divided into at least two adjacent areas of the non-zero pixel point, including: the initial area is divided into multiple sub-areas; wherein the multiple sub-areas include a non-zero area and multiple adjacent areas, the non-zero area includes a non-zero pixel point, and the multiple adjacent areas surround the non-zero area.

[0069] Specifically, the position of the initial region is determined according to the spatial index of the non-zero pixel point, that is, the coordinate position. The initial region may be m×m in size, and the non-zero pixel point may be located at the center. For example, if the initial region is 5×5 in size, the non-zero pixel point may be located at the position of the 3rd row and the 3rd column.

[0070] The initial region is divided into multiple sub-regions, one of which includes only one non-zero pixel, that is, the pixel at the center position forms a sub-region as a non-zero region. Other sub-regions surround the non-zero region, and the other sub-regions are adjacent regions, that is, multiple adjacent regions surround the non-zero region.

[0071] Figure 4The figure is a schematic diagram of the spatial grouping of the initial region. The size of the initial region is 25 pixels, and the size of the adjacent region is 2 pixels or 4 pixels. Around the non-zero pixel in the middle, 8 adjacent regions are divided, that is, there are 9 sub-regions in total. From left to right and from top to bottom, the sub-regions are numbered, starting from 0, and the numbers of the sub-regions are 0, 1, 2, 3, 4, 5, 6, 7, and 8. The sub-region numbered 4 is the non-zero region, and the sub-regions numbered 0, 1, 2, 3, 5, 6, 7, and 8 are adjacent regions.

[0072] The beneficial effect of this setting is that grouping is performed based on preset sizes and grouping strategies, which improves the efficiency of grouping and facilitates finding other non-zero pixels near each non-zero pixel, thereby adjusting the shape of the convolution kernel during subsequent feature extraction and improving the efficiency and accuracy of target detection.

[0073] S304: If it is determined that there is a target pixel in the neighboring area of ​​the non-zero pixel, determine the sequential index of the target pixel; wherein the target pixel represents other non-zero pixels in the image to be processed.

[0074] Exemplarily, each neighboring region includes at least one image pixel point, where the image pixel point is a pixel point in the image to be processed, and the image pixel point in the neighboring region may be a zero pixel point or a non-zero pixel point.

[0075] The center position of the initial area is a non-zero pixel point. It is determined whether there are other non-zero pixels in the initial area except the middle position, that is, whether there are other non-zero pixels in the neighboring area of ​​the middle non-zero pixel point. If it is determined that there are non-zero pixels in the neighboring area, the non-zero pixels in the neighboring area are determined as target pixels. It is possible to find target pixels in each neighboring area, that is, for the non-zero pixel point in the middle position, there may be one or more target pixels corresponding to it, that is, one initial area corresponds to one or more non-zero pixels. If it is determined that there are no non-zero pixels in the neighboring area, then there is no target pixel in the neighboring area.

[0076] Since the target pixel is also a non-zero pixel, the target pixel has its own sequence index to determine the sequence index of each target pixel. For example, if the target pixel is the first non-zero pixel in the image to be processed, the sequence index of the target pixel can be determined to be 0; if the target pixel is the second non-zero pixel in the image to be processed, the sequence index of the target pixel can be determined to be 1. In different initial regions, the sequence index of non-zero pixels at the same spatial index is the same. That is, each spatial index corresponds to a unique sequence index.

[0077] In this embodiment, the neighboring area includes at least one image pixel point; if it is determined that there is a target pixel point in the neighboring area of ​​the non-zero pixel point, then the sequential index of the target pixel point is determined, including: determining the priority of the image pixel point in the neighboring area according to the distance between the non-zero pixel point and the image pixel point in the neighboring area; traversing the image pixel points in the neighboring area according to the priority; if it is determined that the image pixel point is a non-zero pixel point other than the non-zero pixel point, stopping traversing the neighboring area, determining the image pixel point as the target pixel point, and determining the sequential index of the target pixel point.

[0078] Specifically, for each non-zero pixel, after determining the neighboring area of ​​the non-zero pixel, the distance between the non-zero pixel and each image pixel in the neighboring area is determined, and the priority of each image pixel in the neighboring area is determined according to the distance. For example, the image pixel closer to the middle non-zero pixel is set to a higher priority. Each neighboring area must determine the priority of the image pixels contained therein.

[0079] According to the priority, the image pixels in each adjacent area are traversed. For each adjacent area, the traversal can be started from the image pixel with the highest priority to determine whether the image pixel is the target pixel.

[0080] If an image pixel point in the adjacent area is traversed, and the image pixel point is a non-zero pixel point other than the non-zero pixel point in the middle, stop traversing the adjacent area, determine the image pixel point as the target pixel point, and obtain the sequential index of the target pixel point. Continue to traverse the next adjacent area, and determine at most one target pixel point in each adjacent area, that is, for each adjacent pixel point, as long as a target pixel point is traversed, the next adjacent area can be traversed until all adjacent areas in the initial area are traversed.

[0081] The priority of each image pixel can be determined by using the nearest neighbor search or farthest neighbor search sorting rule. The priority can be represented by numbers, and the smaller the number, the higher the priority. Figure 5a This is a priority diagram for the nearest neighbor search. Figure 5b Schematic diagram of the priority of the farthest neighbor search. Figure 5a and Figure 5b In the example, the middle pixel is the non-zero pixel corresponding to the initial area. This pixel does not participate in the priority sorting and its priority can be set to 0. Figure 5a The closer the non-zero pixel is to the middle, the higher the priority. Figure 5b The farther away from the middle non-zero pixel, the higher the priority.

[0082] Construct a mapping table of Nk×Ng according to the priority, and the initial values ​​in the mapping table are all -1. Among them, Nk is the number of sub-regions in the initial region, and Ng is the number of pixels in the sub-region with the largest number of pixels. For example, the initial region is 5×5, there are 9 sub-regions, and there are at most 4 pixels in the sub-region, then the mapping table can be 9 rows and 4 columns, and each row represents a sub-region. The pixels in each sub-region are numbered sequentially and filled into the mapping table according to the priority. The sequential number refers to the number of the pixel points in the initial region. For example, there are 25 pixels in the initial region, and the sequential number can be from 0 to 24. Table 1 is the mapping table for the nearest neighbor search, and Table 2 is the mapping table for the farthest neighbor search.

[0083] Table 1 Mapping table for nearest neighbor search

[0084] 6 5 1 0 7 2 -1 -1 8 3 9 4 11 10 -1 -1 12 -1 -1 -1 13 14 -1 -1 16 21 15 20 17 22 -1 -1 18 19 23 24

[0085] Table 2 Mapping table for the farthest neighbor search

[0086] 0 1 5 6 2 7 -1 -1 4 9 3 8 10 11 -1 -1 12 -1 -1 -1 14 13 -1 -1 20 15 21 16 22 17 -1 -1 24 23 19 18

[0087] Taking the first sub-region in Table 1 as an example, there are four pixels in the first sub-region, which is located in the upper left corner of the initial region. The pixels in the first sub-region are the pixels in the first row and first column, the pixels in the first row and second column, the pixels in the second row and first column, and the pixels in the second row and second column of the initial region. The sequence numbers of these four pixels are 0, 1, 5, and 6, respectively. The sequence number of the image pixel with priority 0 is 6, the sequence number of the image pixel with priority 1 is 5, the sequence number of the image pixel with priority 2 is 1, and the sequence number of the image pixel with priority 3 is 0.

[0088] The beneficial effect of such a setting is that the priority of each pixel in the neighborhood is determined, which facilitates traversal through the priority to find target pixels with higher priority, improves the efficiency of determining target pixels, and thus improves the efficiency of target detection.

[0089] S305 , performing target detection on the object in the image to be processed according to the sparse feature data, the sequential index of each non-zero pixel point, and the sequential index of the target pixel point corresponding to each non-zero pixel point.

[0090] Exemplarily, the sequential indexes of all non-zero pixels are determined, and the sequential indexes of all target pixels corresponding to each non-zero pixel are determined. The target pixels corresponding to the non-zero pixels are the target pixels in all the neighborhood areas of the initial area corresponding to the non-zero pixels.

[0091] According to the sequential index of each non-zero pixel point and the sequential index of each target pixel point corresponding to each non-zero pixel point, the sparse feature data is convolved, that is, feature extraction processing is performed, and the target object in the image to be processed is obtained according to the extracted feature vector. For example, according to the sequential index of the target pixel point corresponding to the non-zero pixel point, the feature vectors of these target pixel points can be extracted from the sparse feature data, and then the feature vectors of these target pixel points corresponding to each non-zero pixel point are fused to obtain the final target feature vector, thereby obtaining the target object in the image to be processed.

[0092] In this embodiment, a preset convolution kernel is used for convolution calculation. For example, a 3×3 convolution kernel can be used. By determining the target pixel point in the adjacent area, the position of the convolution kernel can be transferred to the target pixel point during convolution, thereby expanding the convolution range, changing the shape of the convolution kernel, and improving the ability of feature extraction. It is worth noting that the preset size of the initial area is larger than the size of the convolution kernel. For example, the size of the convolution kernel is 3×3 and the size of the initial area is 5×5.

[0093] In this embodiment, target detection is performed on objects in the image to be processed based on sparse feature data, sequential indexes of each non-zero pixel point, and sequential indexes of target pixels corresponding to each non-zero pixel point, including: determining a neighboring area where the target pixel point corresponding to the non-zero pixel point is located as the target area; associating and storing the area index of the target area with the sequential index of the target pixel point and the sequential index of the non-zero pixel point, respectively; wherein the area index represents the sorting number of the neighboring area; and performing target detection on objects in the image to be processed based on sparse feature data, and the sequential indexes of the non-zero pixel points and the sequential indexes of the target pixel points associated with each area index.

[0094] Specifically, the number of neighboring regions corresponding to each non-zero pixel is the same. For example, each non-zero pixel has 8 neighboring regions. In the initial region, each sub-region is numbered as a region index, and each neighboring region corresponds to a region index. For different non-zero pixels, the region index of the neighboring region is repeated, and the region index in the initial region corresponding to each non-zero pixel is 0 to 8. That is, for each non-zero pixel, the region index of the neighboring region on the upper left is 0, and the region index of the neighboring region directly above is 1.

[0095] If there is a target pixel in the neighboring area of ​​the non-zero pixel, the neighboring area is determined as the target area, and the area index of the target area is determined. For each target area of ​​each non-zero pixel, the area index of the target area is associated with the sequential indexes of all target pixels corresponding to the non-zero pixel and stored, and the area index of the target area is associated with the sequential index of the non-zero pixel.

[0096] Two code tables can be pre-constructed, one for storing the association between the region index of the target region and the sequential indexes of all target pixels corresponding to the non-zero pixels, and the other for storing the association between the region index of the target region and the sequential indexes of the non-zero pixels. The two code tables have the same number of rows, which is the number of neighboring regions. Each row represents a neighboring region. If the neighboring region is the target region, the corresponding sequential index is filled in the row of the neighboring region. In this embodiment, the number of columns of the code table is the number of non-zero pixels, and the initial value of the code table can be set to -1. The number of sequential indexes of the target pixels corresponding to the region index in the code table is consistent with the number of sequential indexes of the corresponding non-zero pixels, and the sequential indexes in the same row of the code table may be repeated.

[0097] For each neighboring area, the sequential index of the non-zero pixel point at the center of the initial area can be determined first, and then the target pixel point that affects the convolution of the non-zero pixel point can be determined. That is, starting from the first non-zero pixel point, the center point of the initial area is placed at the position of the non-zero pixel point, and starting from the first neighboring area in the initial area, it is determined whether there is a target pixel point in the neighboring area. If it is determined that there is a target pixel point in the sixth neighboring area, the sequential index of the target pixel point is stored in the sixth row and first column of the first code table, and the sequential index of the non-zero pixel point at the center of the initial area is stored in the sixth row and first column of the second code table.

[0098] For each region index, determine the sequential indexes of all non-zero pixels associated with the region index and the sequential indexes of all target pixels associated with the region index. Based on the sequential indexes of all non-zero pixels and the sequential indexes of all target pixels associated with each region index, use a preset convolution kernel to perform feature extraction on the sparse feature data and segment the target object from the image to be processed.

[0099] The beneficial effect of this setting is that, by generating two code tables, the relationship between the area index and the sequential index of the target pixel points, as well as the relationship between the area index and the sequential index of the non-zero pixel points can be recorded. Therefore, when performing feature extraction, non-zero pixels in a range larger than the convolution kernel can be found, avoiding limiting the feature extraction capability within the range of the convolution kernel, significantly improving the utilization rate of the convolution kernel and the feature extraction capability, and improving the accuracy of target detection.

[0100] In this embodiment, target detection is performed on objects in the image to be processed based on sparse feature data, sequential indices of non-zero pixels associated with each region index, and sequential indices of target pixels, including: determining input feature data of the region index based on the sparse feature data and sequential indices of target pixels associated with the region index; wherein the input feature data represents a feature vector participating in a convolution operation corresponding to the region index; and performing target detection on objects in the image to be processed based on the input feature data of each region index and sequential indices of non-zero pixels associated with each region index.

[0101] Specifically, the sequential indexes of all target pixels associated with each row of region indexes in the code table are determined, and according to the sequential indexes of all target pixels associated with the region indexes, the feature vector corresponding to the region index is extracted from the sparse feature data as the input feature data of the region index. The input feature data represents the feature vector involved in the convolution operation in the adjacent region represented by the region index. For example, if the sequential index of the target pixel corresponding to the region index is [0, 2], the data of the first and third rows can be extracted from the sparse feature data as the input feature data of the region index.

[0102] Determine the input feature data of all region indexes. If the region index does not have an associated sequential index of target pixels, then there is no corresponding input feature data for the region index. Calculate the target feature vector of the image to be processed based on the input feature data of all region indexes and the sequential index of non-zero pixels associated with these region indexes, and perform target detection on the object in the image to be processed based on the target feature vector. For example, the input feature data can be weighted according to the sequential index of non-zero pixels associated with the region index to obtain the target feature vector.

[0103] The beneficial effect of this setting is that sparse convolution calculation is performed according to the association relationship, so as to achieve targeted feature extraction of each non-zero pixel in each adjacent area, improve the flexibility of convolution, and thus improve the accuracy of target detection.

[0104] In this embodiment, the input feature data of the regional index is determined according to the sequential index of the target pixel points associated with the sparse feature data and the regional index, including: determining the rows to be extracted in the sparse feature data according to the sequential index of the target pixel points associated with the regional index; wherein the rows to be extracted represent the rows extracted from the matrix of the sparse feature data; and obtaining the data of the rows to be extracted from the sparse feature data to constitute the input feature data of the regional index.

[0105] Specifically, a region index may be associated with one or more sequential indexes of target pixels to determine the sequential indexes of all target pixels associated with the region index. For example, the sequential indexes of all target pixels associated with the region index may be expressed in the form of [a1, a2, a3, ...], where a1, a2, and a3 are sequential indexes.

[0106] The sparse feature data can be a matrix vector of multiple rows and columns. The rows to be extracted in the sparse feature data are determined according to the sequential index of the target pixel points associated with the region index. The rows to be extracted represent the rows to be extracted from the matrix of the sparse feature data. For example, if the sequential index of the target pixel points associated with the region index is [0, 1, 0], it means that the rows to be extracted are the first row, the second row, and the first row. That is, the same row in the sparse feature data can be extracted multiple times.

[0107] Get the data of the row to be extracted from the sparse feature data, that is, get the entire row of data to be extracted. According to the data of the row to be extracted, the input feature data of the region index is formed. The data of the row to be extracted can be arranged from top to bottom to generate a new matrix vector as the input feature data. For example, if the rows to be extracted are the first row, the second row, and the first row, then the first row of the sparse feature data can be used as the first row of the input feature data, the second row of the sparse feature data can be used as the second row of the input feature data, and the first row of the sparse feature data can be used as the third row of the input feature data.

[0108] The beneficial effect of this setting is that for each row in the code table, the sequential index of the row is obtained, and the non-zero features of each row are determined according to the sparse feature vector. This enables all non-zero points related to the regional index to be found during convolution calculation, thereby breaking the limitation of the convolution kernel range, and finding non-zero features from the range outside the convolution kernel and within the initial region, thereby improving the ability of feature extraction.

[0109] In this embodiment, target detection is performed on objects in the image to be processed based on the input feature data of each region index and the sequential index of non-zero pixel points associated with each region index, including: determining the output feature data of the region index based on the input feature data of the region index and a preset convolution kernel weight; wherein the output feature data represents a feature vector after the convolution operation; and performing target detection on objects in the image to be processed based on the output feature data of each region index and the sequential index of non-zero pixel points associated with each region index.

[0110] Specifically, a convolution kernel weight is pre-set as a parameter for convolution calculation. Different convolution kernel weights can be set for different region indexes. According to the input feature data of the region index and the convolution kernel weight corresponding to the region index, the output feature data of the region index is determined, and the output feature data represents the feature vector of all target pixels associated with the region index after the convolution operation. The input feature data of the region index can be multiplied by the convolution kernel weight to obtain the output feature data of the region index.

[0111] Determine the sequential indexes of all non-zero pixels associated with each region index, calculate the target feature vector based on the output feature data of all region indexes and the sequential indexes of all non-zero pixels associated with all region indexes, and implement target detection for the object in the processed image. For example, the output feature data can be weightedly calculated based on the sequential indexes of non-zero pixels associated with the region index to obtain the target feature vector.

[0112] The beneficial effect of this setting is that matrix multiplication is performed on the input feature data and the convolution kernel weights to implement convolution calculation on the input feature data and improve the efficiency of target detection.

[0113] In this embodiment, target detection is performed on objects in the image to be processed based on the output feature data of each region index and the sequential index of non-zero pixels associated with each region index, including: for each region index, the output feature data of the region index is added to the same preset all-zero feature vector based on the sequential index of non-zero pixels associated with the region index to obtain a target feature vector of the image to be processed; wherein the target feature vector represents the object in the image to be processed; based on the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

[0114] Specifically, for each region index, the sequential indexes of all non-zero pixels associated with the region index are determined. An all-zero feature vector is pre-set, the number of rows of the all-zero feature vector can be the number of non-zero pixels in the image to be processed, and the number of columns of the all-zero feature vector can be the number of sub-regions. According to the sequential index of the non-zero pixels associated with the region index, the output feature data of the region index is added to the all-zero feature vector. For example, if the sequential index of the non-zero pixels associated with the region index is [0, 1, 2], the first row of the output feature data can be added to the first row of the all-zero feature vector, the second row of the output feature data can be added to the second row of the all-zero feature vector, and the third row of the output feature data can be added to the third row of the all-zero feature vector. In other words, the corresponding row in the output feature data is added to the corresponding row in the all-zero feature vector according to the sequential index of the non-zero pixels.

[0115] The output feature data of all region indexes are added to the same all-zero feature vector. After all the output feature data are added, the all-zero feature vector becomes the target feature vector of the image to be processed. The target feature vector can represent the object to be detected in the image to be processed. According to the target feature vector, the object in the image to be processed is detected and segmented to obtain the object in the image to be processed.

[0116] The beneficial effect of this setting is that the position where feature extraction needs to be performed is changed during convolution calculation through the sequential index of target pixels and the sequential index of non-zero pixels, that is, the shape and position of the convolution kernel are dynamically adjusted, which significantly improves the utilization rate of the convolution kernel and the feature extraction capability, thereby improving the accuracy and efficiency of three-dimensional target detection and segmentation.

[0117] In this embodiment, a neural network model may be preset as a target detection model for executing the image detection method in this embodiment to detect objects from an image. During the training of the model, after obtaining the target feature vector, back propagation may be performed according to the target feature vector to iteratively train the model. For example, model parameters such as the gradient of the output feature data, the non-zero feature gradient, and the convolution kernel weight gradient may be calculated based on the output feature data, and the model parameters may be updated each time the training is performed until the model update is completed.

[0118] In the disclosed embodiment, for an image to be processed of a sparse point cloud, non-zero features in the image to be processed are determined as sparse point cloud data. The spatial index and sequential index of each non-zero pixel in the image to be processed are determined, and the sparse feature data is processed according to the spatial index and sequential index of each non-zero pixel, so as to achieve accurate processing of non-zero pixels at different positions, avoid limitations on the size of the convolution shape during processing, thereby adapting to the structure of irregular point cloud data, and being able to quickly and accurately identify target objects from the image to be processed. By processing non-zero pixels, the amount of computation required for processing point cloud data is greatly reduced, the ability to extract features is improved, and the accuracy and efficiency of target detection are thereby improved.

[0119] Figure 6 A flowchart of an image detection method based on sparse point cloud provided in an embodiment of the present disclosure.

[0120] In this embodiment, determining the spatial index and sequential index of non-zero pixel points in the image to be processed includes: determining the coordinate position of the non-zero pixel point in the image to be processed as the spatial index of the non-zero pixel point; determining the number of non-zero pixel points in the image to be processed; and determining the sequential index of the non-zero pixel point according to the number of non-zero pixel points and the coordinate position of the non-zero pixel point in the image to be processed. This embodiment is based on the above embodiment, such as Figure 6 As shown, the method comprises the following steps:

[0121] S601, obtaining an image to be processed, and determining sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixel points in the image to be processed.

[0122] Exemplarily, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0123] S602: Determine the coordinate position of the non-zero pixel point in the image to be processed, which is the spatial index of the non-zero pixel point.

[0124] Exemplarily, non-zero pixels are located in the image to be processed, and each non-zero pixel corresponds to its own coordinate position, indicating the position of the non-zero pixel in the image to be processed. The coordinate position of the non-zero pixel is used as the spatial index of the non-zero pixel. That is, the spatial index is in the coordinate form of (x, y).

[0125] S603: Determine the number of non-zero pixels in the image to be processed.

[0126] Exemplarily, the image to be processed includes a plurality of non-zero pixels, and the number of the non-zero pixels in the image to be processed is determined.

[0127] S604: Determine the sequential index of the non-zero pixels according to the number of non-zero pixels and the coordinate positions of the non-zero pixels in the image to be processed.

[0128] Exemplarily, the non-zero pixels are sorted according to the number of non-zero pixels and the coordinate positions of the non-zero pixels in the image to be processed, and the sorted numbers are used as the sequential indexes of the non-zero pixels. For example, starting from the first pixel in the first row of the image to be processed, sequential indexes can be assigned to each non-zero pixel in turn from left to right and from top to bottom. Figure 7 Schematic diagram of the sequential index of non-zero pixels. Figure 7 There are 6 non-zero pixels in total, with sequential indexes from 0 to 5.

[0129] In this embodiment, the coordinate positions of non-zero pixels are determined as spatial indexes, and the order indexes are determined based on the number of non-zero pixels, starting from 0, to achieve orderly management of non-zero pixels, avoid confusion in pixel calculation, and improve target detection accuracy.

[0130] S605 , performing target detection on the object in the image to be processed according to the sparse feature data and the spatial index and the sequential index of each non-zero pixel point.

[0131] Exemplarily, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0132] In the disclosed embodiment, for an image to be processed of a sparse point cloud, non-zero features in the image to be processed are determined as sparse point cloud data. The spatial index and sequential index of each non-zero pixel in the image to be processed are determined, and the sparse feature data is processed according to the spatial index and sequential index of each non-zero pixel, so as to achieve accurate processing of non-zero pixels at different positions, avoid limitations on the size of the convolution shape during processing, thereby adapting to the structure of irregular point cloud data, and being able to quickly and accurately identify target objects from the image to be processed. By processing non-zero pixels, the amount of computation required for processing point cloud data is greatly reduced, the ability to extract features is improved, and the accuracy and efficiency of target detection are thereby improved.

[0133] Figure 8 The following is a block diagram of a sparse point cloud-based image detection device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 8 The image detection device 800 based on sparse point cloud includes: an image acquisition unit 801, an index determination unit 802 and an image detection unit 803.

[0134] The image acquisition unit 801 is used to acquire an image to be processed and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed;

[0135] An index determination unit 802 is used to determine a spatial index and a sequential index of a non-zero pixel in the image to be processed; wherein the spatial index represents a coordinate position of the non-zero pixel in the image to be processed, and the sequential index represents a pixel ordering number of the non-zero pixel in the image to be processed;

[0136] The image detection unit 803 is used to perform target detection on the object in the image to be processed according to the sparse feature data and the spatial index and sequential index of each non-zero pixel point.

[0137] Fig. 9 A structural block diagram of an image detection device based on sparse point cloud provided in an embodiment of the present disclosure, such as Fig. 9 As shown, the image detection device 900 based on sparse point cloud includes an image acquisition unit 901, an index determination unit 902 and an image detection unit 903, wherein the image detection unit 903 includes a neighborhood determination module 9031, a target determination module 9032 and an image detection module 9033.

[0138] A neighborhood determination module 9031 is used to determine the neighborhood of the non-zero pixel in the image to be processed according to the spatial index of the non-zero pixel; wherein the neighborhood represents a preset area range adjacent to the non-zero pixel in the image to be processed;

[0139] A target determination module 9032 is used to determine the sequential index of the target pixel point if it is determined that there is a target pixel point in the neighboring area of ​​the non-zero pixel point; wherein the target pixel point represents other non-zero pixel points in the image to be processed;

[0140] The image detection module 9033 is used to perform target detection on the object in the image to be processed according to the sparse feature data, the sequential index of each non-zero pixel point, and the sequential index of the target pixel point corresponding to each non-zero pixel point.

[0141] In one example, the neighborhood determination module 9031 includes:

[0142] An initial determination submodule, configured to determine an initial region of a preset size according to the spatial index of the non-zero pixel point; wherein the non-zero pixel point is located in the initial region;

[0143] The region division submodule is used to divide the initial region into at least two adjacent regions of the non-zero pixel point.

[0144] In one example, the non-zero pixel point is located at the center of the initial area; the area division submodule is specifically used to:

[0145] The initial area is divided into a plurality of sub-areas; wherein the plurality of sub-areas include a non-zero area and a plurality of adjacent areas, the non-zero area includes a non-zero pixel point, and the plurality of adjacent areas surround the non-zero area.

[0146] In one example, the neighboring area includes at least one image pixel point; the target determination module 9032 includes:

[0147] a priority determination submodule, configured to determine the priority of the image pixel points in the adjacent area according to the distance between the non-zero pixel point and the image pixel points in the adjacent area;

[0148] A pixel traversal submodule, used for traversing the image pixels in the adjacent area according to the priority;

[0149] The target determination submodule is used to stop traversing the adjacent area if it is determined that the image pixel point is a non-zero pixel point other than the non-zero pixel point, determine the image pixel point as a target pixel point, and determine the sequential index of the target pixel point.

[0150] In one example, the image detection module 9033 includes:

[0151] A region determination submodule is used to determine the neighboring region where the target pixel point corresponding to the non-zero pixel point is located as the target region;

[0152] An associated storage submodule, used for associating and storing the region index of the target region with the sequential index of the target pixel points and the sequential index of the non-zero pixel points, respectively; wherein the region index represents the sorting number of the adjacent regions;

[0153] The target detection submodule is used to perform target detection on the object in the image to be processed according to the sparse feature data, the sequential index of the non-zero pixel points associated with each area index and the sequential index of the target pixel points.

[0154] In one example, the target detection submodule is specifically used to:

[0155] Determine input feature data of the region index according to the sequential index of the target pixel points associated with the sparse feature data and the region index; wherein the input feature data represents a feature vector participating in the convolution operation corresponding to the region index;

[0156] According to the input feature data of each region index and the sequential index of non-zero pixel points associated with each region index, target detection is performed on the object in the image to be processed.

[0157] In one example, the target detection submodule is specifically used to:

[0158] Determine the rows to be extracted in the sparse feature data according to the sequential index of the target pixel points associated with the region index; wherein the rows to be extracted represent rows extracted from the matrix of the sparse feature data;

[0159] The data of the row to be extracted is obtained from the sparse feature data to form the input feature data of the region index.

[0160] In one example, the target detection submodule is specifically used to:

[0161] Determining output feature data of the region index according to the input feature data of the region index and a preset convolution kernel weight; wherein the output feature data represents a feature vector after a convolution operation;

[0162] According to the output feature data of each region index and the sequential index of non-zero pixel points associated with each region index, target detection is performed on the object in the image to be processed.

[0163] In one example, the target detection submodule is specifically used to:

[0164] For each region index, according to the sequential index of non-zero pixel points associated with the region index, the output feature data of the region index are all added to the same preset all-zero feature vector to obtain the target feature vector of the image to be processed; wherein the target feature vector represents the object in the image to be processed;

[0165] According to the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

[0166] In one example, the index determination unit 902 includes:

[0167] A space determination module, used to determine the coordinate position of a non-zero pixel point in the image to be processed, which is the spatial index of the non-zero pixel point;

[0168] A quantity determination module, used to determine the number of non-zero pixel points in the image to be processed;

[0169] The sequence determination module is used to determine the sequence index of the non-zero pixel points according to the number of non-zero pixel points in the image to be processed and the coordinate positions of the non-zero pixel points.

[0170] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0171] Fig.10 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, such as Fig.10 As shown, the electronic device 1000 includes: at least one processor 1002; and a memory 1001 communicatively connected to the at least one processor 1002; wherein the memory stores instructions executable by the at least one processor 1002, and the instructions are executed by the at least one processor 1002 so that the at least one processor 1002 can execute the sparse point cloud-based image detection method disclosed in the present invention.

[0172] The electronic device 1000 further includes a receiver 1003 and a transmitter 1004. The receiver 1003 is used to receive instructions and data sent by other devices, and the transmitter 1004 is used to send instructions and data to external devices.

[0173] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0174] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0175] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0176] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0177] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0178] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as an image detection method based on a sparse point cloud. For example, in some embodiments, the image detection method based on a sparse point cloud may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the image detection method based on the sparse point cloud described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the sparse point cloud-based image detection method in any other appropriate manner (eg, by means of firmware).

[0179] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0180] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0181] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0183] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0184] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0185] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0186] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image detection method based on sparse point cloud, comprising: Acquire an image to be processed, and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed; Determine the spatial index and sequential index of the non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel point sorting number of the non-zero pixel point in the image to be processed; According to the sparse feature data, and the spatial index and the sequential index of each non-zero pixel point, target detection is performed on the object in the image to be processed.

2. The method according to claim 1, wherein: The method of performing target detection on the object in the image to be processed according to the sparse feature data and the spatial index and the sequential index of each non-zero pixel point comprises: Determine, according to the spatial index of the non-zero pixel point, a neighboring area of ​​the non-zero pixel point in the image to be processed; wherein the neighboring area represents a preset area range adjacent to the non-zero pixel point in the image to be processed; If it is determined that there is a target pixel point in the neighboring area of ​​the non-zero pixel point, then determining the sequential index of the target pixel point; wherein the target pixel point represents other non-zero pixel points in the image to be processed; According to the sparse feature data, the sequential index of each non-zero pixel point, and the sequential index of the target pixel point corresponding to each non-zero pixel point, target detection is performed on the object in the image to be processed.

3. The method according to claim 2, wherein: The step of determining the neighboring area of ​​the non-zero pixel in the image to be processed according to the spatial index of the non-zero pixel includes: Determine an initial region of a preset size according to the spatial index of the non-zero pixel point; wherein the non-zero pixel point is located in the initial region; The initial area is divided into at least two adjacent areas of the non-zero pixel point.

4. The method according to claim 3, wherein: The non-zero pixel point is located at the center of the initial area; and the initial area is divided into at least two adjacent areas of the non-zero pixel point, including: The initial area is divided into a plurality of sub-areas; wherein the plurality of sub-areas include a non-zero area and a plurality of adjacent areas, the non-zero area includes a non-zero pixel point, and the plurality of adjacent areas surround the non-zero area.

5. The method according to any one of claims 2 to 4, wherein: The neighboring area includes at least one image pixel point; if it is determined that there is a target pixel point in the neighboring area of ​​the non-zero pixel point, determining the sequential index of the target pixel point includes: Determine the priority of the image pixel points in the adjacent area according to the distance between the non-zero pixel point and the image pixel points in the adjacent area; According to the priority, traverse the image pixels in the neighboring area; If it is determined that the image pixel point is a non-zero pixel point other than the non-zero pixel point, then the traversal of the adjacent area is stopped, the image pixel point is determined as a target pixel point, and a sequential index of the target pixel point is determined.

6. The method according to any one of claims 2 to 5, wherein: The method of performing target detection on the object in the image to be processed according to the sparse feature data, the sequential index of each non-zero pixel point, and the sequential index of the target pixel point corresponding to each non-zero pixel point comprises: Determine the neighboring area where the target pixel point corresponding to the non-zero pixel point is located as the target area; The region index of the target region is stored in association with the sequential index of the target pixel points and the sequential index of the non-zero pixel points respectively; wherein the region index represents the sorting number of the adjacent regions; According to the sparse feature data, the sequential indexes of non-zero pixels associated with each region index and the sequential indexes of target pixels, target detection is performed on the object in the image to be processed.

7. The method according to claim 6, wherein: The method of performing target detection on the object in the image to be processed according to the sparse feature data, the sequential indexes of the non-zero pixels associated with each region index, and the sequential indexes of the target pixels comprises: Determine input feature data of the region index according to the sequential index of the target pixel points associated with the sparse feature data and the region index; wherein the input feature data represents a feature vector participating in the convolution operation corresponding to the region index; According to the input feature data of each region index and the sequential index of non-zero pixels associated with each region index, target detection is performed on the object in the image to be processed.

8. The method according to claim 7, wherein: The step of determining the input feature data of the region index according to the sequential index of the target pixel points associated with the sparse feature data and the region index comprises: Determine the rows to be extracted in the sparse feature data according to the sequential index of the target pixel points associated with the region index; wherein the rows to be extracted represent rows extracted from the matrix of the sparse feature data; The data of the row to be extracted is obtained from the sparse feature data to form the input feature data of the region index.

9. The method according to claim 7 or 8, wherein: The method of performing target detection on the object in the image to be processed according to the input feature data of each region index and the sequential index of the non-zero pixel points associated with each region index comprises: Determining output feature data of the region index according to the input feature data of the region index and a preset convolution kernel weight; wherein the output feature data represents a feature vector after a convolution operation; According to the output feature data of each region index and the sequential index of non-zero pixels associated with each region index, target detection is performed on the object in the image to be processed.

10. The method according to claim 9, wherein: The method of performing target detection on the object in the image to be processed according to the output feature data of each region index and the sequential index of non-zero pixels associated with each region index comprises: For each region index, according to the sequential index of non-zero pixel points associated with the region index, the output feature data of the region index are all added to the same preset all-zero feature vector to obtain the target feature vector of the image to be processed; wherein the target feature vector represents the object in the image to be processed; According to the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

11. The method according to any one of claims 1 to 10, wherein: The step of determining the spatial index and the sequential index of the non-zero pixel points in the image to be processed includes: Determine the coordinate position of the non-zero pixel point in the image to be processed, which is the spatial index of the non-zero pixel point; Determining the number of non-zero pixels in the image to be processed; The sequential indexes of the non-zero pixels are determined according to the number of non-zero pixels in the image to be processed and the coordinate positions of the non-zero pixels.

12. An image detection device based on sparse point cloud, comprising: An image acquisition unit, used to acquire an image to be processed and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed; An index determination unit, used to determine the spatial index and sequential index of the non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel point sorting number of the non-zero pixel point in the image to be processed; The image detection unit is used to perform target detection on the object in the image to be processed according to the sparse feature data and the spatial index and sequential index of each non-zero pixel point.

13. The device according to claim 12, wherein: The image detection unit comprises: A neighborhood determination module, used to determine the neighborhood of the non-zero pixel in the image to be processed according to the spatial index of the non-zero pixel; wherein the neighborhood represents a preset area range adjacent to the non-zero pixel in the image to be processed; A target determination module, used for determining the sequential index of the target pixel point if it is determined that there is a target pixel point in the neighboring area of ​​the non-zero pixel point; wherein the target pixel point represents other non-zero pixel points in the image to be processed; The image detection module is used to perform target detection on the object in the image to be processed according to the sparse feature data, the sequential index of each non-zero pixel point, and the sequential index of the target pixel point corresponding to each non-zero pixel point.

14. The device according to claim 13, wherein: The neighborhood determination module comprises: An initial determination submodule, configured to determine an initial region of a preset size according to the spatial index of the non-zero pixel point; wherein the non-zero pixel point is located in the initial region; The region division submodule is used to divide the initial region into at least two adjacent regions of the non-zero pixel point.

15. The device according to claim 14, wherein: The non-zero pixel point is located at the center of the initial area; the area division submodule is specifically used to: The initial area is divided into a plurality of sub-areas; wherein the plurality of sub-areas include a non-zero area and a plurality of adjacent areas, the non-zero area includes a non-zero pixel point, and the plurality of adjacent areas surround the non-zero area.

16. The device according to any one of claims 13 to 15, wherein: The adjacent area includes at least one image pixel point; The target determination module comprises: a priority determination submodule, configured to determine the priority of the image pixel points in the adjacent area according to the distance between the non-zero pixel point and the image pixel points in the adjacent area; A pixel traversal submodule, used for traversing the image pixels in the adjacent area according to the priority; The target determination submodule is used to stop traversing the adjacent area if it is determined that the image pixel point is a non-zero pixel point other than the non-zero pixel point, determine the image pixel point as a target pixel point, and determine the sequential index of the target pixel point.

17. The device according to any one of claims 13 to 16, wherein: The image detection module comprises: A region determination submodule is used to determine the neighboring region where the target pixel point corresponding to the non-zero pixel point is located as the target region; An associated storage submodule, used for associating and storing the region index of the target region with the sequential index of the target pixel points and the sequential index of the non-zero pixel points, respectively; wherein the region index represents the sorting number of the adjacent regions; The target detection submodule is used to perform target detection on the object in the image to be processed according to the sparse feature data, the sequential index of the non-zero pixel points associated with each area index and the sequential index of the target pixel points.

18. The device according to claim 17, wherein: The target detection submodule is specifically used for: Determine input feature data of the region index according to the sequential index of the target pixel points associated with the sparse feature data and the region index; wherein the input feature data represents a feature vector participating in the convolution operation corresponding to the region index; According to the input feature data of each region index and the sequential index of non-zero pixel points associated with each region index, target detection is performed on the object in the image to be processed.

19. The device according to claim 18, wherein: The target detection submodule is specifically used for: Determine the rows to be extracted in the sparse feature data according to the sequential index of the target pixel points associated with the region index; wherein the rows to be extracted represent rows extracted from the matrix of the sparse feature data; The data of the row to be extracted is obtained from the sparse feature data to form the input feature data of the region index.

20. The device according to claim 18 or 19, wherein: The target detection submodule is specifically used for: Determining output feature data of the region index according to the input feature data of the region index and a preset convolution kernel weight; wherein the output feature data represents a feature vector after a convolution operation; According to the output feature data of each region index and the sequential index of non-zero pixel points associated with each region index, target detection is performed on the object in the image to be processed.

21. The device according to claim 20, wherein: The target detection submodule is specifically used for: For each region index, according to the sequential index of non-zero pixel points associated with the region index, the output feature data of the region index are all added to the same preset all-zero feature vector to obtain the target feature vector of the image to be processed; wherein the target feature vector represents the object in the image to be processed; According to the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

22. The device according to any one of claims 12 to 21, wherein: The index determination unit includes: A space determination module, used to determine the coordinate position of a non-zero pixel point in the image to be processed, which is the spatial index of the non-zero pixel point; A quantity determination module, used to determine the number of non-zero pixel points in the image to be processed; The sequence determination module is used to determine the sequence index of the non-zero pixel points according to the number of non-zero pixel points in the image to be processed and the coordinate positions of the non-zero pixel points.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.

25. A computer program product, wherein: The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 11 when being executed by a processor.

Images