Image detection method, device, equipment and storage medium based on sparse point cloud

Abstract

The present disclosure provides an image detection method, apparatus, device, and storage medium based on sparse point clouds, which relates to the field of artificial intelligence technology, particularly computer vision, deep learning, and other technical fields, and can be applied to scenarios such as autonomous driving and robot navigation. The solution includes: obtaining an image to be processed and determining sparse feature data of the image to be processed; the image to be processed is sparse point cloud data, and the sparse feature data represents the non-zero pixels in the image to be processed; determining the spatial index and sequential index of the non-zero pixels; the spatial index represents the coordinate position of the non-zero pixels in the image to be processed, and the sequential index represents the pixel ordering number of the non-zero pixels in the image to be processed; and performing target detection on the image to be processed based on the sparse feature data, the spatial index, and the sequential index of each non-zero pixel. By determining the spatial index and sequential index of the non-zero pixels, the limitations on the shape of the convolution kernel are reduced, and the accuracy of target detection is improved.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as computer vision and deep learning in the field of artificial intelligence, and can be applied to scenarios such as autonomous driving and robot navigation. In particular, it relates to an image detection method, device, equipment and storage medium based on sparse point cloud. Background Art

[0002] With the rapid development of 3D technology, 3D data has been widely used in fields such as autonomous driving, robot navigation, and industrial inspection. 3D data can be in the form of point clouds, and the distribution of point clouds can represent the distribution of objects in the environment.

[0003] In sparse point cloud data, due to the relatively scattered distribution of point clouds, traditional convolutional neural networks have problems such as large computational complexity, high memory usage, and low convolution kernel utilization when performing image processing, which affects the detection accuracy and efficiency of objects in the image. Summary of the Invention

[0004] The present disclosure provides an image detection method, apparatus, device and storage medium based on sparse point cloud.

[0005] According to a first aspect of the present disclosure, there is provided an image detection method based on a sparse point cloud, comprising:

[0006] Acquire an image to be processed, and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed;

[0007] Determine the spatial index and sequential index of the non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel order number of the non-zero pixel point in the image to be processed;

[0008] Target detection is performed on objects in the image to be processed according to the sparse feature data and the spatial index and sequential index of each non-zero pixel point.

[0009] According to a second aspect of the present disclosure, there is provided an image detection device based on a sparse point cloud, comprising:

[0010] An image acquisition unit, configured to acquire an image to be processed and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed;

[0011] An index determination unit, configured to determine a spatial index and a sequential index of a non-zero pixel in the image to be processed; wherein the spatial index represents a coordinate position of the non-zero pixel in the image to be processed, and the sequential index represents a pixel order number of the non-zero pixel in the image to be processed;

[0012] The image detection unit is used to perform target detection on the object in the image to be processed based on the sparse feature data and the spatial index and sequential index of each non-zero pixel point.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively coupled to the at least one processor;

[0016] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the present disclosure when executed by a processor.

[0019] According to the technology disclosed in the present invention, the flexibility of convolution on sparse point cloud data is improved, thereby improving the accuracy and efficiency of target detection.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0022] Figure 1 1 is a flow chart of an image detection method based on sparse point cloud provided in accordance with an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of non-zero pixels in an image to be processed according to an embodiment of the present disclosure;

[0024] Figure 3 1 is a flow chart of an image detection method based on sparse point cloud provided in accordance with an embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of spatial grouping of initial areas provided according to an embodiment of the present disclosure;

[0026] Figure 5a 1 is a schematic diagram of the priority of the nearest neighbor search provided according to an embodiment of the present disclosure;

[0027] Figure 5b 1 is a schematic diagram of the priority of the farthest neighbor search provided according to an embodiment of the present disclosure;

[0028] Figure 6 1 is a flow chart of an image detection method based on sparse point cloud provided in accordance with an embodiment of the present disclosure;

[0029] Figure 7 is a schematic diagram of sequential indexing of non-zero pixels provided according to an embodiment of the present disclosure;

[0030] Figure 8 is a structural block diagram of an image detection device based on sparse point cloud provided according to an embodiment of the present disclosure;

[0031] Figure 9 is a structural block diagram of an image detection device based on sparse point cloud provided according to an embodiment of the present disclosure;

[0032] Figure 10 is a block diagram of an electronic device for implementing the sparse point cloud-based image detection method according to an embodiment of the present disclosure;

[0033] Figure 11 It is a block diagram of an electronic device used to implement the sparse point cloud-based image detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] With the rapid development of 3D acquisition technology, 3D data captured by sensors such as LiDAR and RGB-D (Red, Green, Blue-Depth) cameras has found widespread application in fields such as autonomous driving, robotic navigation, and industrial inspection. However, traditional convolutional neural networks (CNNs) face challenges with processing sparse point cloud data, including high computational complexity, high memory usage, and low kernel utilization. Improving the efficiency and accuracy of sparse convolutions, particularly in 3D object detection and segmentation tasks, has become a hot topic of research.

[0036] Submanifold sparse convolution can reduce unnecessary computation by performing convolution operations directly on sparse point clouds. However, the fixed shape of the convolution kernel makes it difficult to adapt to the irregular structure of sparse point clouds, limiting its feature extraction capabilities. Current submanifold sparse convolution methods can increase the receptive field by increasing the size of the convolution kernel. However, a larger receptive field can affect the accuracy and efficiency of object detection. Therefore, dynamically adjusting the shape of the convolution kernel to better fit the structure of sparse point clouds is key to improving sparse convolution performance.

[0037] The present disclosure provides an image detection method, apparatus, device, and storage medium based on sparse point clouds, which are applied to computer vision, deep learning, and other technical fields in the field of artificial intelligence. They can be applied to scenarios such as autonomous driving and robot navigation to improve the flexibility of convolution and thereby improve the accuracy and efficiency of target detection.

[0038] It should be noted that the data in this embodiment is not data for a specific user and cannot reflect the personal information of a specific user. It should be noted that the images in this embodiment are from a public data set.

[0039] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0040] In order to enable readers to have a deeper understanding of the implementation principle of this disclosure, the following Figures 1-11 The embodiment is further refined.

[0041] Figure 1 FIG. 1 is a flow chart of an image detection method based on sparse point clouds according to an embodiment of the present disclosure. The method can be performed by an image detection device based on sparse point clouds. Figure 1 As shown, the method includes the following steps:

[0042] S101 , obtaining an image to be processed, and determining sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed.

[0043] For example, the image to be processed can be acquired by a sensor such as a lidar or RGB-D sensor. The image to be processed can be point cloud data. In this embodiment, the image to be processed is sparse point cloud data. The point cloud in sparse point cloud data is sparsely or irregularly distributed, with some locations being zero pixels and some locations being non-zero pixels. That is, the pixels in the image to be processed include both zero pixels and non-zero pixels. A zero pixel refers to a pixel at which no valid image pixel information exists, while a non-zero pixel refers to a pixel at which image pixel information exists.

[0044] After obtaining the image to be processed, a convolution process can be performed on the image to be processed according to a preset convolution layer to obtain 2D sparse features of the image to be processed as sparse feature data. In this embodiment, the sparse feature data can represent non-zero pixels in the image to be processed, that is, the sparse feature data represents the non-zero features of the image to be processed. In this embodiment, the convolution process for obtaining the 2D sparse features is not specifically limited.

[0045] The sparse feature data may be an N×C matrix vector, where N is the number of non-zero features, that is, the number of non-zero pixels, and C is the feature dimension. The feature dimension may be the number of channels of the image feature, for example, the number of RGB channels. Figure 2 is a schematic diagram of non-zero pixels in the image to be processed. Figure 2 Indicated by black squares, Figure 2 There are 6 non-zero pixels in , that is, N is 6.

[0046] S102. Determine the spatial index and sequential index of non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel order number of the non-zero pixel point in the image to be processed.

[0047] For example, each pixel in an image has its own coordinate position, representing the distribution of the pixel in the image. The coordinate position of a non-zero pixel in the image to be processed is determined as the spatial index of the non-zero pixel. That is, the spatial index represents the coordinate position of the non-zero pixel in the image to be processed.

[0048] For each non-zero pixel, determine the sequential index of the non-zero pixel. The sequential index represents the pixel order number of the non-zero pixel in the image to be processed, and the sequential index can be an integer starting from 0. That is, starting from 0, each non-zero pixel is numbered. For example, if there are 6 non-zero pixels in the image to be processed, the non-zero pixels can be sorted starting from 0 according to the order of the non-zero pixels in the image to be processed, and the sequential indices of the non-zero pixels are 0, 1, 2, 3, 4, and 5 respectively. Starting from the first row of pixels, the non-zero pixels can be searched for in the order of the pixels from left to right, and the non-zero pixels can be numbered.

[0049] S103 : Perform target detection on the object in the image to be processed according to the sparse feature data and the spatial index and sequential index of each non-zero pixel point.

[0050] For example, there may be one or more objects in the image to be processed, and the point cloud distribution of the locations of the objects in the image to be processed will be relatively dense. For example, the image to be processed is a road image, and the objects in the image to be processed may be vehicles or pedestrians on the road.

[0051] Sparse feature data can be processed based on the spatial and sequential indices of all non-zero pixels in the image to be processed. For example, object-related information can be extracted from the image to be processed based on the spatial and sequential indices of each non-zero pixel, and the resulting feature vector can be used as the target feature vector. In other words, the target feature vector can represent the object in the image to be processed. Based on the target feature vector, the image to be processed is segmented to identify the object in the image to be processed, thus completing object detection in the image to be processed. The detected object is the object in the image to be processed.

[0052] A convolution kernel of a preset size can be pre-set. Based on the spatial and sequential indices of non-zero pixels, sparse feature data can be processed using this kernel. For example, the preset convolution kernel is a sparse convolution, specifically a submanifold sparse convolution. The convolution kernel can be translated based on the spatial indices of the non-zero pixels, moving the kernel to each non-zero pixel. For example, the center of the kernel can be moved to the coordinates of the non-zero pixel. If the convolution kernel is 5×5, the kernel position at row 3 and column 3 can be sequentially placed at each non-zero pixel. For each non-zero pixel, the kernel is convolved with the non-zero features of the non-zero pixels near the kernel based on their sequential indices. For example, non-zero pixels located outside the kernel and adjacent to the kernel can be identified. During convolution, the zero pixels in the kernel are not processed, but the adjacent non-zero pixels are processed, thereby changing the shape and size of the kernel. The processing results at each position are combined to obtain the target feature vector. For example, the processing results of the convolution kernel at each non-zero pixel point can be added to obtain the target feature vector.

[0053] In the embodiment of the present disclosure, for an image to be processed of a sparse point cloud, non-zero features in the image to be processed are determined as sparse point cloud data. The spatial index and sequential index of each non-zero pixel in the image to be processed are determined, and the sparse feature data is processed according to the spatial index and sequential index of each non-zero pixel, so as to achieve accurate processing of non-zero pixels at different positions, avoid limitations on the size of the convolution shape during processing, thereby adapting to the structure of irregular point cloud data, and being able to quickly and accurately identify target objects from the image to be processed. By processing non-zero pixels, the amount of computation required for processing point cloud data is greatly reduced, the ability to extract features is improved, and the accuracy and efficiency of target detection are thereby improved.

[0054] Figure 3 A flowchart of an image detection method based on sparse point clouds provided in an embodiment of the present disclosure.

[0055] In this embodiment, target detection is performed on objects in the image to be processed based on sparse feature data, and the spatial index and sequential index of each non-zero pixel point, including: determining the neighboring area of ​​the non-zero pixel point in the image to be processed based on the spatial index of the non-zero pixel point; wherein the neighboring area represents a preset area range adjacent to the non-zero pixel point in the image to be processed; if it is determined that there is a target pixel point in the neighboring area of ​​the non-zero pixel point, then determining the sequential index of the target pixel point; wherein the target pixel point represents other non-zero pixel points in the image to be processed; target detection is performed on objects in the image to be processed based on the sparse feature data, the sequential index of each non-zero pixel point, and the sequential index of the target pixel points corresponding to each non-zero pixel point.

[0056] This embodiment is based on the above embodiment. Figure 3 As shown, the method includes the following steps:

[0057] S301 , obtaining an image to be processed, and determining sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed.

[0058] For example, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0059] S302, determining the spatial index and sequential index of the non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel order number of the non-zero pixel point in the image to be processed.

[0060] For example, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0061] S303 , determining a neighboring region of the non-zero pixel in the image to be processed according to a spatial index of the non-zero pixel; wherein the neighboring region represents a preset region range adjacent to the non-zero pixel in the image to be processed.

[0062] Exemplarily, the spatial index can be a representation of coordinates, for example, the spatial index is (x, y), which represents the position of the non-zero pixel in the image to be processed. For each non-zero pixel, based on the spatial index of the non-zero pixel, a neighboring area near the position of the non-zero pixel is determined from the image to be processed. The neighboring area is the neighborhood of the non-zero pixel, that is, a preset area range adjacent to the non-zero pixel. For example, the area of ​​two pixels above the non-zero pixel can be used as the neighboring area of ​​the non-zero pixel.

[0063] A non-zero pixel may correspond to one or more neighboring regions. For example, a neighboring region may be defined above, below, left, and right of the non-zero pixel. Multiple different preset region ranges may be preset, i.e., the sizes of the neighboring regions of a non-zero pixel may vary. For example, the neighboring region directly above the non-zero pixel may be two pixels in size, while the neighboring region to the upper left of the non-zero pixel may be four pixels in size.

[0064] In this embodiment, the neighboring areas of the non-zero pixel points in the image to be processed are determined based on the spatial index of the non-zero pixel points, including: determining an initial area of ​​a preset size based on the spatial index of the non-zero pixel points; wherein the non-zero pixel points are located in the initial area; and dividing the initial area into at least two neighboring areas of the non-zero pixel points.

[0065] Specifically, the size of the initial region is pre-set, and an initial region of the preset size is obtained based on the spatial index of the non-zero pixel and the preset size. For example, a 5×5 initial region is preset. Each non-zero pixel corresponds to an initial region of the preset size, and the non-zero pixel is located within the corresponding initial region. For example, if there are 6 non-zero pixels in the image to be processed, there are 6 initial regions, and there is a 5×5 initial region at the location of each non-zero pixel, and the non-zero pixel is a pixel in the 5×5.

[0066] The initial region is spatially grouped and divided into multiple parts. The divided parts can be used as the neighborhood of non-zero pixels, and multiple neighborhoods of non-zero pixels can be obtained. For example, the pixels in the initial region other than non-zero pixels can be grouped to obtain multiple neighborhoods, and each neighborhood can include one or more pixels.

[0067] The beneficial effect of this setting is that the initial area is spatially grouped, which makes it easier to find non-zero pixels from different spaces, and facilitates finding non-zero pixels in different spaces during the subsequent convolution processing of feature extraction, thereby realizing the change of the convolution kernel shape, improving the ability of feature extraction, and improving the efficiency and accuracy of target detection.

[0068] In this embodiment, the non-zero pixel point is located at the center of the initial area; dividing the initial area into at least two adjacent areas of the non-zero pixel point includes: dividing the initial area into multiple sub-areas; wherein the multiple sub-areas include a non-zero area and multiple adjacent areas, the non-zero area includes a non-zero pixel point, and the multiple adjacent areas surround the non-zero area.

[0069] Specifically, the location of the initial region is determined based on the spatial index, or coordinate position, of the non-zero pixel. The initial region can be m×m in size, and the non-zero pixel can be located at the center. For example, if the initial region is 5×5 in size, the non-zero pixel can be located at the 3rd row and 3rd column.

[0070] The initial region is divided into multiple subregions. One of the subregions contains only one non-zero pixel, that is, the pixel at the center is a subregion in itself, which is the non-zero region. The other subregions surround the non-zero region, and the other subregions are called adjacent regions. In other words, multiple adjacent regions surround the non-zero region.

[0071] Figure 4Figure 2 shows the spatial grouping of the initial region. The initial region is 25 pixels in size, and the adjacent regions can be 2 or 4 pixels in size. Eight adjacent regions are divided around the central non-zero pixel, resulting in a total of nine sub-regions. These sub-regions are numbered from left to right and top to bottom, starting with 0, and are numbered 0, 1, 2, 3, 4, 5, 6, 7, and 8, respectively. Sub-region numbered 4 is the non-zero region, and sub-regions numbered 0, 1, 2, 3, 5, 6, 7, and 8 are adjacent regions.

[0072] The beneficial effect of this setting is that grouping is performed based on preset sizes and grouping strategies, which improves the efficiency of grouping and facilitates finding other non-zero pixels near each non-zero pixel, thereby adjusting the shape of the convolution kernel during subsequent feature extraction and improving the efficiency and accuracy of target detection.

[0073] S304: If it is determined that a target pixel exists in the neighboring area of ​​the non-zero pixel, determine a sequential index of the target pixel; wherein the target pixel represents other non-zero pixels in the image to be processed.

[0074] Exemplarily, each neighboring region includes at least one image pixel point, where the image pixel point is a pixel point in the image to be processed. The image pixel point in the neighboring region may be a zero pixel point or a non-zero pixel point.

[0075] The center position of the initial area is a non-zero pixel point. It is determined whether there are other non-zero pixels in the initial area besides the middle position, that is, whether there are other non-zero pixels in the adjacent area of ​​the middle non-zero pixel point. If it is determined that there are non-zero pixels in the adjacent area, the non-zero pixels in the adjacent area are determined as the target pixels. It is possible to find the target pixel point in each adjacent area, that is, for the non-zero pixel point in the middle position, there may be one or more target pixels corresponding to it. In other words, one initial area corresponds to one or more non-zero pixels. If it is determined that there are no non-zero pixels in the adjacent area, then there is no target pixel point in the adjacent area.

[0076] Since the target pixel is also a non-zero pixel, it has its own sequence index. For example, if the target pixel is the first non-zero pixel in the image to be processed, its sequence index can be determined as 0; if the target pixel is the second non-zero pixel in the image to be processed, its sequence index can be determined as 1. In different initial regions, the sequence index of non-zero pixels at the same spatial index is the same. In other words, each spatial index corresponds to a unique sequence index.

[0077] In this embodiment, the neighboring area includes at least one image pixel point; if it is determined that a target pixel point exists in the neighboring area of ​​a non-zero pixel point, a sequential index of the target pixel point is determined, including: determining the priority of the image pixel point in the neighboring area based on the distance between the non-zero pixel point and the image pixel point in the neighboring area; traversing the image pixel points in the neighboring area based on the priority; if it is determined that the image pixel point is a non-zero pixel point other than the non-zero pixel point, stopping traversing the neighboring area, determining the image pixel point as the target pixel point, and determining the sequential index of the target pixel point.

[0078] Specifically, for each non-zero pixel, after determining the non-zero pixel's neighboring area, the distance between the non-zero pixel and each image pixel in the neighboring area is determined. The priority of each image pixel in the neighboring area is determined based on the distance. For example, image pixels closer to the central non-zero pixel are assigned a higher priority. Priorities for the image pixels contained in each neighboring area are determined separately.

[0079] According to the priority, the image pixels in each adjacent area are traversed. For each adjacent area, the traversal can be started from the image pixel with the highest priority to determine whether the image pixel is the target pixel.

[0080] If an image pixel point in the adjacent area is found, and this image pixel point is a non-zero pixel point other than the middle non-zero pixel point, then the traversal of the adjacent area is stopped, and the image pixel point is determined as the target pixel point, and the sequential index of the target pixel point is obtained. The traversal of the next adjacent area is continued, and at most one target pixel point is determined in each adjacent area. That is, for each adjacent pixel point, as long as a target pixel point is traversed, the traversal of the next adjacent area can be started, until all the adjacent areas in the initial area are traversed.

[0081] The priority of each image pixel can be determined by using a sorting rule of nearest neighbor search or farthest neighbor search. The priority can be represented by a number, where a smaller number indicates a higher priority. Figure 5a Schematic diagram of the priority of nearest neighbor search. Figure 5b Schematic diagram of the priority of the farthest neighbor search. Figure 5a and Figure 5b In the example, the middle pixel is the non-zero pixel corresponding to the initial area. This pixel does not participate in the priority sorting and its priority can be set to 0. Figure 5a The closer the non-zero pixel is to the middle, the higher the priority. Figure 5b The farther away from the middle non-zero pixel, the higher the priority.

[0082] Construct an Nk×Ng mapping table based on the priority, and the initial values ​​in the mapping table are all -1. Among them, Nk is the number of sub-regions in the initial region, and Ng is the number of pixels in the sub-region with the largest number of pixels. For example, if the initial region is 5×5, there are 9 sub-regions, and there are at most 4 pixels in a sub-region, then the mapping table can be 9 rows and 4 columns, with each row representing a sub-region. The pixels in each sub-region are numbered sequentially and filled into the mapping table according to the priority. The sequential number refers to the number of the pixel in the initial region. For example, if there are 25 pixels in the initial region, the sequential number can be from 0 to 24. Table 1 is the mapping table for the nearest neighbor search, and Table 2 is the mapping table for the farthest neighbor search.

[0083] Table 1 Mapping table for nearest neighbor search

[0084] 6 5 1 0 7 2 -1 -1 8 3 9 4 11 10 -1 -1 12 -1 -1 -1 13 14 -1 -1 16 21 15 20 17 22 -1 -1 18 19 23 24

[0085] Table 2 Mapping table for farthest neighbor search

[0086] 0 1 5 6 2 7 -1 -1 4 9 3 8 10 11 -1 -1 12 -1 -1 -1 14 13 -1 -1 20 15 21 16 22 17 -1 -1 24 23 19 18

[0087] Taking the first subregion in Table 1 as an example, there are four pixels in the first subregion, located in the upper left corner of the initial region. The pixels in the first subregion are the pixel in the first row and first column, the pixel in the first row and second column, the pixel in the second row and first column, and the pixel in the second row and second column of the initial region. These four pixels are numbered 0, 1, 5, and 6, respectively. The image pixel with priority 0 is numbered 6, the image pixel with priority 1 is numbered 5, the image pixel with priority 2 is numbered 1, and the image pixel with priority 3 is numbered 0.

[0088] The beneficial effect of this setting is that it determines the priority of each pixel in the neighborhood, facilitates traversal through priority, finds target pixels with higher priority, improves the efficiency of determining target pixels, and thus improves the efficiency of target detection.

[0089] S305 , performing target detection on the object in the image to be processed according to the sparse feature data, the sequential index of each non-zero pixel point, and the sequential index of the target pixel point corresponding to each non-zero pixel point.

[0090] For example, the sequential indices of all non-zero pixels are determined, and the sequential indices of all target pixels corresponding to each non-zero pixel are determined. The target pixels corresponding to the non-zero pixels are the target pixels in all neighboring areas of the initial area corresponding to the non-zero pixels.

[0091] Based on the sequential indices of each non-zero pixel point and the sequential indices of each target pixel point corresponding to each non-zero pixel point, a convolution calculation is performed on the sparse feature data, i.e., feature extraction processing is performed. The target object in the image to be processed is obtained based on the extracted feature vector. For example, feature vectors of the target pixels corresponding to the non-zero pixels can be extracted from the sparse feature data based on the sequential indices of the target pixels corresponding to the non-zero pixels. The feature vectors of the target pixels corresponding to the non-zero pixels are then fused to obtain a final target feature vector, thereby obtaining the target object in the image to be processed.

[0092] In this embodiment, a preset convolution kernel is used for convolution calculation. For example, a 3×3 convolution kernel can be used. By determining the target pixel point in the adjacent area, the position of the convolution kernel can be transferred to the target pixel point during convolution, thereby expanding the convolution range, changing the shape of the convolution kernel, and improving the feature extraction capability. It is worth noting that the preset size of the initial area is larger than the size of the convolution kernel. For example, the size of the convolution kernel is 3×3 and the size of the initial area is 5×5.

[0093] In this embodiment, target detection is performed on objects in the image to be processed based on sparse feature data, sequential indexes of each non-zero pixel point, and sequential indexes of target pixels corresponding to each non-zero pixel point, including: determining the adjacent area where the target pixel point corresponding to the non-zero pixel point is located as the target area; associating and storing the area index of the target area with the sequential index of the target pixel point and the sequential index of the non-zero pixel point, respectively; wherein the area index represents the sorting number of the adjacent area; target detection is performed on objects in the image to be processed based on sparse feature data, and the sequential indexes of the non-zero pixel points and the sequential index of the target pixel point associated with each area index.

[0094] Specifically, each non-zero pixel corresponds to the same number of neighboring regions. For example, each non-zero pixel has 8 neighboring regions. Each subregion in the initial region is numbered as a region index, and each neighboring region has a corresponding region index. For different non-zero pixels, the region index of the neighboring regions is repeated. The region index in the initial region corresponding to each non-zero pixel is 0 to 8. That is, for each non-zero pixel, the region index of the neighboring region to the upper left is 0, and the region index of the neighboring region directly above is 1.

[0095] If a target pixel exists in the neighborhood of a non-zero pixel, the neighborhood is determined as the target area, and the area index of the target area is determined. For each target area of ​​each non-zero pixel, the area index of the target area is associated and stored with the sequential indexes of all target pixels corresponding to the non-zero pixel, and the area index of the target area is associated and stored with the sequential index of the non-zero pixel.

[0096] Two code tables can be pre-built: one for storing the association between the region index of the target region and the sequential indices of all target pixels corresponding to non-zero pixels, and one for storing the association between the region index of the target region and the sequential indices of non-zero pixels. Both code tables have the same number of rows, which is the number of neighboring regions. Each row represents a neighboring region. If a neighboring region is the target region, the corresponding sequential index is entered in the row of that neighboring region. In this embodiment, the number of columns in the code table is the same as the number of non-zero pixels, and the initial value of the code table can be set to -1. The number of sequential indices of target pixels corresponding to the region index in the code table is the same as the number of sequential indices of the corresponding non-zero pixels. Sequential indices in the same row of the code table can be repeated.

[0097] For each adjacent region, the sequential index of the non-zero pixel at the center of the initial region can be determined first, and then the target pixel that affects the convolution of the non-zero pixel can be determined. That is, starting from the first non-zero pixel, the center point of the initial region is placed at the position of the non-zero pixel, and starting from the first adjacent region in the initial region, it is determined whether the target pixel exists in the adjacent region. If it is determined that the target pixel exists in the sixth adjacent region, the sequential index of the target pixel is stored in the sixth row and first column of the first code table, and the sequential index of the non-zero pixel at the center of the initial region is stored in the sixth row and first column of the second code table.

[0098] For each region index, determine the sequential indices of all non-zero pixels associated with that region index, as well as the sequential indices of all target pixels associated with that region index. Based on the sequential indices of all non-zero pixels and all target pixels associated with each region index, use a preset convolution kernel to extract features from the sparse feature data and segment the target object from the image to be processed.

[0099] The beneficial effect of this setting is that by generating two code tables, the relationship between the region index and the sequential index of the target pixel points, as well as the relationship between the region index and the sequential index of the non-zero pixel points can be recorded. Therefore, when performing feature extraction, non-zero pixels in a range larger than the convolution kernel can be found, avoiding limiting the feature extraction capability to the range of the convolution kernel, significantly improving the utilization rate of the convolution kernel and the feature extraction capability, and improving the accuracy of target detection.

[0100] In this embodiment, target detection is performed on objects in the image to be processed based on sparse feature data, sequential indices of non-zero pixels associated with each region index, and sequential indices of target pixels, including: determining input feature data of the region index based on the sparse feature data and sequential indices of target pixels associated with the region index; wherein the input feature data represents a feature vector participating in a convolution operation corresponding to the region index; and performing target detection on objects in the image to be processed based on the input feature data of each region index and sequential indices of non-zero pixels associated with each region index.

[0101] Specifically, the sequential indices of all target pixels associated with each row of region indexes in the code table are determined. Based on the sequential indices of all target pixels associated with the region indexes, the feature vector corresponding to the region index is extracted from the sparse feature data as the input feature data for the region index. The input feature data represents the feature vectors participating in the convolution operation within the adjacent region represented by the region index. For example, if the sequential indices of the target pixels corresponding to the region index are [0, 2], then the data of the first and third rows of the sparse feature data can be extracted from the sparse feature data as the input feature data for the region index.

[0102] Determine the input feature data for all region indexes. If a region index does not have an associated sequential index of a target pixel, then no corresponding input feature data exists for that region index. Calculate a target feature vector for the image to be processed based on the input feature data for all region indexes and the sequential indices of the non-zero pixels associated with those region indexes. Detect objects in the image to be processed based on the target feature vector. For example, the target feature vector can be obtained by performing a weighted calculation on the input feature data based on the sequential indices of the non-zero pixels associated with the region indexes.

[0103] The beneficial effect of this setting is that sparse convolution calculation is performed according to the correlation relationship, and targeted feature extraction is achieved for each non-zero pixel point in each adjacent area, which improves the flexibility of convolution and thus improves the accuracy of target detection.

[0104] In this embodiment, the input feature data of the region index is determined based on the sequential index of the target pixel points associated with the sparse feature data and the region index, including: determining the rows to be extracted in the sparse feature data based on the sequential index of the target pixel points associated with the region index; wherein the rows to be extracted represent rows extracted from the matrix of the sparse feature data; and obtaining data of the rows to be extracted from the sparse feature data to constitute the input feature data of the region index.

[0105] Specifically, a region index can be associated with one or more sequential indices of target pixels to determine the sequential indices of all target pixels associated with the region index. For example, the sequential indices of all target pixels associated with the region index can be expressed in the form of [a1, a2, a3, ...], where a1, a2, and a3 are sequential indices.

[0106] Sparse feature data can be a matrix vector with multiple rows and columns. The rows to be extracted from the sparse feature data are determined based on the sequential indices of the target pixels associated with the region index. These rows represent the rows to be extracted from the sparse feature data matrix. For example, if the sequential indices of the target pixels associated with the region index are [0, 1, 0], then the rows to be extracted are the first row, the second row, and the first row. In other words, the same row in the sparse feature data can be extracted multiple times.

[0107] Obtain the data for the row to be extracted from the sparse feature data, that is, obtain the entire row of data to be extracted. Based on the data for the row to be extracted, compose the input feature data for the region index. The data for the row to be extracted can be arranged from top to bottom to generate a new matrix vector, which serves as the input feature data. For example, if the rows to be extracted are the first, second, and first rows, then the first row of the sparse feature data can be used as the first row of the input feature data, the second row of the sparse feature data as the second row of the input feature data, and the first row of the sparse feature data as the third row of the input feature data.

[0108] The beneficial effect of this setup is that for each row in the code table, the sequential index of the row is obtained, and the non-zero features of each row are determined based on the sparse feature vector. This allows all non-zero points associated with the region index to be found during convolution calculations, thus breaking the limitations of the convolution kernel range and being able to find non-zero features from outside the convolution kernel and within the initial region, improving feature extraction capabilities.

[0109] In this embodiment, target detection is performed on objects in the image to be processed based on the input feature data of each region index and the sequential index of non-zero pixel points associated with each region index, including: determining the output feature data of the region index based on the input feature data of the region index and a preset convolution kernel weight; wherein the output feature data represents a feature vector after the convolution operation; and performing target detection on objects in the image to be processed based on the output feature data of each region index and the sequential index of non-zero pixel points associated with each region index.

[0110] Specifically, convolution kernel weights are pre-set as parameters for the convolution calculation. Different convolution kernel weights can be set for different region indexes. Based on the input feature data of the region index and the convolution kernel weight corresponding to the region index, the output feature data of the region index is determined. The output feature data represents the feature vector of all target pixels associated with the region index after the convolution operation. The input feature data of the region index can be multiplied by the convolution kernel weight to obtain the output feature data of the region index.

[0111] Determine the sequential indices of all non-zero pixels associated with each region index, and calculate a target feature vector based on the output feature data of all region indexes and the sequential indices of all non-zero pixels associated with all region indexes, thereby detecting objects in the processed image. For example, the target feature vector can be obtained by performing a weighted calculation on the output feature data based on the sequential indices of the non-zero pixels associated with the region indexes.

[0112] The beneficial effect of this setting is that matrix multiplication is performed on the input feature data and the convolution kernel weights to realize the convolution calculation of the input feature data and improve the efficiency of target detection.

[0113] In this embodiment, target detection is performed on objects in the image to be processed based on the output feature data of each region index and the sequential index of non-zero pixel points associated with each region index, including: for each region index, the output feature data of the region index is added to the same preset all-zero feature vector based on the sequential index of non-zero pixel points associated with the region index to obtain a target feature vector of the image to be processed; wherein the target feature vector represents the object in the image to be processed; based on the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

[0114] Specifically, for each region index, the sequential indexes of all non-zero pixels associated with the region index are determined. An all-zero feature vector is pre-set, the number of rows of the all-zero feature vector can be the number of non-zero pixels in the image to be processed, and the number of columns of the all-zero feature vector can be the number of sub-regions. According to the sequential index of the non-zero pixels associated with the region index, the output feature data of the region index is added to the all-zero feature vector. For example, if the sequential index of the non-zero pixels associated with the region index is [0, 1, 2], then the first row of the output feature data can be added to the first row of the all-zero feature vector, the second row of the output feature data can be added to the second row of the all-zero feature vector, and the third row of the output feature data can be added to the third row of the all-zero feature vector. In other words, the corresponding row in the output feature data is added to the corresponding row in the all-zero feature vector according to the sequential index of the non-zero pixels.

[0115] The output feature data of all region indices is summed with the same all-zero feature vector. After all the output feature data are added, the all-zero feature vector becomes the target feature vector of the image to be processed. The target feature vector can represent the object to be detected in the image to be processed. Based on the target feature vector, the object in the image to be processed is detected and segmented to obtain the object in the image to be processed.

[0116] The beneficial effect of this setting is that the position where feature extraction needs to be performed is changed during convolution calculation through the sequential index of target pixels and the sequential index of non-zero pixels, that is, the shape and position of the convolution kernel are dynamically adjusted, which significantly improves the utilization rate of the convolution kernel and the feature extraction capability, thereby improving the accuracy and efficiency of three-dimensional target detection and segmentation.

[0117] In this embodiment, a neural network model can be preset as a target detection model for executing the image detection method of this embodiment to detect objects from images. During the model training process, after obtaining the target feature vector, backpropagation can be performed based on the target feature vector to iteratively train the model. For example, based on the output feature data, model parameters such as the gradient of the output feature data, the non-zero feature gradient, and the convolution kernel weight gradient can be calculated. The model parameters can be updated with each training until the model update is complete.

[0118] In the embodiment of the present disclosure, for an image to be processed of a sparse point cloud, non-zero features in the image to be processed are determined as sparse point cloud data. The spatial index and sequential index of each non-zero pixel in the image to be processed are determined, and the sparse feature data is processed according to the spatial index and sequential index of each non-zero pixel, so as to achieve accurate processing of non-zero pixels at different positions, avoid limitations on the size of the convolution shape during processing, thereby adapting to the structure of irregular point cloud data, and being able to quickly and accurately identify target objects from the image to be processed. By processing non-zero pixels, the amount of computation required for processing point cloud data is greatly reduced, the ability to extract features is improved, and the accuracy and efficiency of target detection are thereby improved.

[0119] Figure 6 A flowchart of an image detection method based on sparse point clouds provided in an embodiment of the present disclosure.

[0120] In this embodiment, determining the spatial index and sequential index of non-zero pixel points in the image to be processed includes: determining the coordinate position of the non-zero pixel point in the image to be processed as the spatial index of the non-zero pixel point; determining the number of non-zero pixel points in the image to be processed; and determining the sequential index of the non-zero pixel points based on the number of non-zero pixel points and the coordinate position of the non-zero pixel points in the image to be processed. This embodiment is based on the above embodiment, such as Figure 6 As shown, the method includes the following steps:

[0121] S601 , obtaining an image to be processed, and determining sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed.

[0122] For example, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0123] S602: Determine the coordinate positions of non-zero pixels in the image to be processed, which are spatial indexes of the non-zero pixels.

[0124] For example, non-zero pixels are located in the image to be processed, and each non-zero pixel has its own corresponding coordinate position, which represents the position of the non-zero pixel in the image to be processed. The coordinate position of the non-zero pixel is used as the spatial index of the non-zero pixel. That is, the spatial index is in the coordinate form of (x, y).

[0125] S603: Determine the number of non-zero pixels in the image to be processed.

[0126] Exemplarily, the image to be processed includes a plurality of non-zero pixels, and the number of the non-zero pixels in the image to be processed is determined.

[0127] S604 : Determine the sequential index of the non-zero pixels according to the number of the non-zero pixels and the coordinate positions of the non-zero pixels in the image to be processed.

[0128] For example, the non-zero pixels are sorted based on the number and coordinate positions of the non-zero pixels in the image to be processed, and the sorted numbers are used as sequential indices of the non-zero pixels. For example, starting from the first pixel in the first row of the image to be processed, sequential indices can be assigned to each non-zero pixel in turn from left to right and from top to bottom. Figure 7 Schematic diagram of the sequential index of non-zero pixels. Figure 7 There are 6 non-zero pixels in total, with sequential indexes from 0 to 5.

[0129] In this embodiment, the coordinate positions of non-zero pixel points are determined as spatial indexes, and the order indexes are determined based on the number of non-zero pixel points, starting from 0, to achieve orderly management of non-zero pixel points, avoid confusion in pixel point calculation, and improve target detection accuracy.

[0130] S605 : Perform target detection on the object in the image to be processed according to the sparse feature data and the spatial index and sequential index of each non-zero pixel point.

[0131] For example, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0132] In the embodiment of the present disclosure, for an image to be processed of a sparse point cloud, non-zero features in the image to be processed are determined as sparse point cloud data. The spatial index and sequential index of each non-zero pixel in the image to be processed are determined, and the sparse feature data is processed according to the spatial index and sequential index of each non-zero pixel, so as to achieve accurate processing of non-zero pixels at different positions, avoid limitations on the size of the convolution shape during processing, thereby adapting to the structure of irregular point cloud data, and being able to quickly and accurately identify target objects from the image to be processed. By processing non-zero pixels, the amount of computation required for processing point cloud data is greatly reduced, the ability to extract features is improved, and the accuracy and efficiency of target detection are thereby improved.

[0133] Figure 8 This is a structural block diagram of an image detection device based on sparse point cloud provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 8 The image detection device 800 based on sparse point cloud includes: an image acquisition unit 801, an index determination unit 802 and an image detection unit 803.

[0134] The image acquisition unit 801 is configured to acquire an image to be processed and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed;

[0135] An index determination unit 802 is configured to determine a spatial index and a sequential index of a non-zero pixel in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel in the image to be processed, and the sequential index represents the pixel order of the non-zero pixel in the image to be processed;

[0136] The image detection unit 803 is configured to perform target detection on the object in the image to be processed according to the sparse feature data and the spatial index and sequential index of each non-zero pixel point.

[0137] Figure 9 This is a structural block diagram of an image detection device based on sparse point cloud provided by an embodiment of the present disclosure, such as Figure 9 As shown, the image detection device 900 based on sparse point cloud includes an image acquisition unit 901, an index determination unit 902 and an image detection unit 903, wherein the image detection unit 903 includes a neighborhood determination module 9031, a target determination module 9032 and an image detection module 9033.

[0138] A neighborhood determination module 9031 is configured to determine a neighborhood of the non-zero pixel in the image to be processed based on the spatial index of the non-zero pixel; wherein the neighborhood represents a preset area range adjacent to the non-zero pixel in the image to be processed;

[0139] A target determination module 9032 is configured to determine a sequential index of a target pixel if it is determined that a target pixel exists in a neighborhood of the non-zero pixel; wherein the target pixel represents other non-zero pixels in the image to be processed;

[0140] The image detection module 9033 is used to perform target detection on the object in the image to be processed based on the sparse feature data, the sequential index of each non-zero pixel point, and the sequential index of the target pixel point corresponding to each non-zero pixel point.

[0141] In one example, the neighborhood determination module 9031 includes:

[0142] an initial determination submodule, configured to determine an initial region of a preset size according to a spatial index of the non-zero pixel point; wherein the non-zero pixel point is located in the initial region;

[0143] The region division submodule is configured to divide the initial region into at least two adjacent regions of the non-zero pixel point.

[0144] In one example, the non-zero pixel point is located at the center of the initial area; the area division submodule is specifically used to:

[0145] The initial area is divided into multiple sub-areas; wherein the multiple sub-areas include a non-zero area and multiple adjacent areas, the non-zero area includes a non-zero pixel point, and the multiple adjacent areas surround the non-zero area.

[0146] In one example, the neighboring area includes at least one image pixel; the target determination module 9032 includes:

[0147] a priority determination submodule, configured to determine the priority of the image pixel points in the adjacent area according to the distance between the non-zero pixel point and the image pixel points in the adjacent area;

[0148] A pixel traversal submodule, configured to traverse the image pixels in the adjacent area according to the priority;

[0149] The target determination submodule is used to stop traversing the adjacent area if it is determined that the image pixel point is a non-zero pixel point other than the non-zero pixel point, determine the image pixel point as a target pixel point, and determine the sequential index of the target pixel point.

[0150] In one example, the image detection module 9033 includes:

[0151] A region determination submodule is used to determine the adjacent region where the target pixel corresponding to the non-zero pixel is located as the target region;

[0152] an associated storage submodule, configured to associate and store the region index of the target region with the sequential index of the target pixel point and the sequential index of the non-zero pixel point, respectively; wherein the region index represents the sorting number of the adjacent regions;

[0153] The target detection submodule is used to perform target detection on the object in the image to be processed according to the sparse feature data, the sequential index of the non-zero pixel points associated with each region index, and the sequential index of the target pixel points.

[0154] In one example, the target detection submodule is specifically used to:

[0155] Determine input feature data of the region index according to the sequential index of the target pixel points associated with the sparse feature data and the region index; wherein the input feature data represents a feature vector participating in the convolution operation corresponding to the region index;

[0156] Target detection is performed on objects in the image to be processed according to the input feature data of each region index and the sequential index of non-zero pixels associated with each region index.

[0157] In one example, the target detection submodule is specifically used to:

[0158] Determining a row to be extracted from the sparse feature data according to a sequential index of the target pixel points associated with the region index; wherein the row to be extracted represents a row extracted from the matrix of the sparse feature data;

[0159] The data of the row to be extracted is obtained from the sparse feature data to form the input feature data of the region index.

[0160] In one example, the target detection submodule is specifically used to:

[0161] Determining output feature data of the region index based on the input feature data of the region index and a preset convolution kernel weight; wherein the output feature data represents a feature vector after the convolution operation;

[0162] Target detection is performed on objects in the image to be processed according to the output feature data of each region index and the sequential index of non-zero pixels associated with each region index.

[0163] In one example, the target detection submodule is specifically used to:

[0164] For each region index, adding the output feature data of the region index to the same preset all-zero feature vector according to the sequential index of the non-zero pixel points associated with the region index to obtain a target feature vector of the image to be processed; wherein the target feature vector represents the object in the image to be processed;

[0165] According to the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

[0166] In one example, the index determination unit 902 includes:

[0167] A spatial determination module, configured to determine the coordinate positions of non-zero pixels in the image to be processed, which are spatial indexes of the non-zero pixels;

[0168] a quantity determination module, configured to determine the number of non-zero pixel features in the image to be processed;

[0169] The sequence determination module is used to determine the sequence index of the non-zero pixel points according to the number of non-zero pixel points in the image to be processed and the coordinate positions of the non-zero pixel points.

[0170] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0171] Figure 10 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, such as Figure 10 As shown, the electronic device 1000 includes: at least one processor 1002; and a memory 1001 communicatively connected to the at least one processor 1002; wherein the memory stores instructions that can be executed by the at least one processor 1002, and the instructions are executed by the at least one processor 1002 to enable the at least one processor 1002 to execute the sparse point cloud-based image detection method disclosed in the present invention.

[0172] The electronic device 1000 further includes a receiver 1003 and a transmitter 1004. The receiver 1003 is used to receive instructions and data sent by other devices, and the transmitter 1004 is used to send instructions and data to external devices.

[0173] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0174] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0175] Figure 11 A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0176] like Figure 11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0177] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0178] The computing unit 1101 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the sparse point cloud-based image detection method. For example, in some embodiments, the sparse point cloud-based image detection method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the sparse point cloud-based image detection method described above can be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to execute the sparse point cloud-based image detection method in any other appropriate manner (for example, by means of firmware).

[0179] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0180] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0181] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0182] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0183] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0184] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0185] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0186] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An image detection method based on sparse point clouds, comprising: Acquire an image to be processed, and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed; Determine the spatial index and sequential index of the non-zero pixel points in the image to be processed; wherein the spatial index represents the coordinate position of the non-zero pixel point in the image to be processed, and the sequential index represents the pixel order number of the non-zero pixel point in the image to be processed; Performing target detection on an object in the image to be processed according to the sparse feature data and the spatial index and the sequential index of each non-zero pixel point, including: Determine an adjacent area where the target pixel corresponding to the non-zero pixel is located as the target area; The region index of the target region is associated with the sequential index of the target pixel point and the sequential index of the non-zero pixel point and stored; wherein the region index represents the sorting number of the adjacent regions; Performing target detection on objects in the image to be processed according to the sparse feature data, the sequential indexes of non-zero pixels associated with each region index, and the sequential indexes of target pixels; By using the sequential index of the target pixel points and the sequential index of the non-zero pixel points, the position where feature extraction needs to be performed is changed during convolution calculation to dynamically adjust the convolution kernel.

2. The method according to claim 1, wherein The method further comprises: performing target detection on the object in the image to be processed according to the sparse feature data and the spatial index and the sequential index of each non-zero pixel point. Determining a neighboring area of ​​the non-zero pixel in the image to be processed according to the spatial index of the non-zero pixel; wherein the neighboring area represents a preset area range adjacent to the non-zero pixel in the image to be processed; If it is determined that there is a target pixel point in the neighborhood of the non-zero pixel point, then the sequential index of the target pixel point is determined; wherein the target pixel point represents other non-zero pixel points in the image to be processed.

3. The method according to claim 2, wherein: The determining, based on the spatial index of the non-zero pixel point, a neighboring area of ​​the non-zero pixel point in the image to be processed includes: Determining an initial region of a preset size according to the spatial index of the non-zero pixel point; wherein the non-zero pixel point is located in the initial region; The initial area is divided into at least two adjacent areas of the non-zero pixel point.

4. The method according to claim 3, wherein: The non-zero pixel point is located at the center of the initial area; and the dividing the initial area into at least two adjacent areas of the non-zero pixel point includes: The initial area is divided into a plurality of sub-areas; wherein the plurality of sub-areas include a non-zero area and a plurality of adjacent areas, the non-zero area includes a non-zero pixel point, and the plurality of adjacent areas surround the non-zero area.

5. The method according to any one of claims 2 to 4, wherein The adjacent area includes at least one image pixel point; if it is determined that a target pixel point exists in the adjacent area of ​​the non-zero pixel point, determining the sequential index of the target pixel point includes: determining the priority of the image pixel points in the adjacent area according to the distance between the non-zero pixel point and the image pixel points in the adjacent area; Traversing the image pixels in the adjacent area according to the priority; If it is determined that the image pixel point is a non-zero pixel point other than the non-zero pixel point, then the traversal of the adjacent area is stopped, the image pixel point is determined as the target pixel point, and the sequential index of the target pixel point is determined.

6. The method according to claim 1, wherein The method of performing target detection on an object in the image to be processed according to the sparse feature data, the sequential indexes of non-zero pixels associated with each region index, and the sequential indexes of target pixels includes: Determine input feature data of the region index according to the sequential index of the target pixel points associated with the sparse feature data and the region index; wherein the input feature data represents a feature vector participating in the convolution operation corresponding to the region index; Target detection is performed on objects in the image to be processed according to the input feature data of each region index and the sequential index of non-zero pixels associated with each region index.

7. The method according to claim 6, wherein: The step of determining the input feature data of the region index according to the sequential index of the target pixel points associated with the sparse feature data and the region index comprises: Determining a row to be extracted from the sparse feature data according to a sequential index of the target pixel points associated with the region index; wherein the row to be extracted represents a row extracted from the matrix of the sparse feature data; The data of the row to be extracted is obtained from the sparse feature data to form the input feature data of the region index.

8. The method according to claim 6 or 7, wherein: The method of detecting an object in the image to be processed according to the input feature data of each region index and the sequential index of non-zero pixels associated with each region index includes: Determining output feature data of the region index based on the input feature data of the region index and a preset convolution kernel weight; wherein the output feature data represents a feature vector after the convolution operation; Target detection is performed on objects in the image to be processed according to the output feature data of each region index and the sequential index of non-zero pixels associated with each region index.

9. The method according to claim 8, wherein The method of detecting an object in the image to be processed based on the output feature data of each region index and the sequential index of non-zero pixels associated with each region index includes: For each region index, adding the output feature data of the region index to the same preset all-zero feature vector according to the sequential index of the non-zero pixel points associated with the region index to obtain a target feature vector of the image to be processed; wherein the target feature vector represents the object in the image to be processed; According to the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

10. The method according to any one of claims 1-4, 6-7, and 9, wherein Determining the spatial index and the sequential index of the non-zero pixel points in the image to be processed includes: Determine the coordinate position of a non-zero pixel point in the image to be processed as the spatial index of the non-zero pixel point; Determining the number of non-zero pixels in the image to be processed; Determine the sequential indexes of the non-zero pixels according to the number of non-zero pixels in the image to be processed and the coordinate positions of the non-zero pixels.

11. An image detection device based on sparse point cloud, comprising: An image acquisition unit, configured to acquire an image to be processed and determine sparse feature data in the image to be processed; wherein the image to be processed is sparse point cloud data, and the sparse feature data represents non-zero pixels in the image to be processed; An index determination unit, configured to determine a spatial index and a sequential index of a non-zero pixel in the image to be processed; wherein the spatial index represents a coordinate position of the non-zero pixel in the image to be processed, and the sequential index represents a pixel order number of the non-zero pixel in the image to be processed; An image detection unit, configured to perform target detection on an object in the image to be processed based on the sparse feature data and the spatial index and sequential index of each non-zero pixel point; The image detection unit includes: A region determination submodule is used to determine the adjacent region where the target pixel corresponding to the non-zero pixel is located as the target region; an associated storage submodule, configured to associate and store the region index of the target region with the sequential index of the target pixel point and the sequential index of the non-zero pixel point, respectively; wherein the region index represents the sorting number of the adjacent regions; The target detection submodule is used to perform target detection on objects in the image to be processed based on the sparse feature data and the sequential indexes of the non-zero pixels and the sequential indexes of the target pixels associated with each region index; through the sequential indexes of the target pixels and the sequential indexes of the non-zero pixels, the position where feature extraction needs to be performed is changed during the convolution calculation to dynamically adjust the convolution kernel.

12. The device according to claim 11, wherein The image detection unit further includes: a neighborhood determination module, configured to determine a neighborhood of the non-zero pixel in the image to be processed based on a spatial index of the non-zero pixel; wherein the neighborhood represents a preset area range adjacent to the non-zero pixel in the image to be processed; The target determination module is used to determine the sequential index of the target pixel point if it is determined that there is a target pixel point in the adjacent area of ​​the non-zero pixel point; wherein the target pixel point represents other non-zero pixels in the image to be processed.

13. The device according to claim 12, wherein The neighborhood determination module includes: an initial determination submodule, configured to determine an initial region of a preset size according to a spatial index of the non-zero pixel point; wherein the non-zero pixel point is located in the initial region; The region division submodule is configured to divide the initial region into at least two adjacent regions of the non-zero pixel point.

14. The device according to claim 13, wherein The non-zero pixel point is located at the center of the initial area; the area division submodule is specifically used to: The initial area is divided into multiple sub-areas; wherein the multiple sub-areas include a non-zero area and multiple adjacent areas, the non-zero area includes a non-zero pixel point, and the multiple adjacent areas surround the non-zero area.

15. The device according to any one of claims 12 to 14, wherein: The adjacent area includes at least one image pixel point; The target determination module includes: a priority determination submodule, configured to determine the priority of the image pixel points in the adjacent area according to the distance between the non-zero pixel point and the image pixel points in the adjacent area; A pixel traversal submodule, configured to traverse the image pixels in the adjacent area according to the priority; The target determination submodule is used to stop traversing the adjacent area if it is determined that the image pixel point is a non-zero pixel point other than the non-zero pixel point, determine the image pixel point as a target pixel point, and determine the sequential index of the target pixel point.

16. The device according to claim 11, wherein The target detection submodule is specifically used to: Determine input feature data of the region index according to the sequential index of the target pixel points associated with the sparse feature data and the region index; wherein the input feature data represents a feature vector participating in the convolution operation corresponding to the region index; Target detection is performed on objects in the image to be processed according to the input feature data of each region index and the sequential index of non-zero pixels associated with each region index.

17. The device according to claim 16, wherein The target detection submodule is specifically used to: Determining a row to be extracted from the sparse feature data according to a sequential index of the target pixel points associated with the region index; wherein the row to be extracted represents a row extracted from the matrix of the sparse feature data; The data of the row to be extracted is obtained from the sparse feature data to form the input feature data of the region index.

18. The device according to claim 16 or 17, wherein The target detection submodule is specifically used to: Determining output feature data of the region index based on the input feature data of the region index and a preset convolution kernel weight; wherein the output feature data represents a feature vector after the convolution operation; Target detection is performed on objects in the image to be processed according to the output feature data of each region index and the sequential index of non-zero pixels associated with each region index.

19. The device according to claim 18, wherein The target detection submodule is specifically used to: For each region index, adding the output feature data of the region index to the same preset all-zero feature vector according to the sequential index of the non-zero pixel points associated with the region index to obtain a target feature vector of the image to be processed; wherein the target feature vector represents the object in the image to be processed; According to the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

20. The device according to any one of claims 11-14, 16-17, and 19, wherein: The index determination unit includes: A spatial determination module, configured to determine the coordinate positions of non-zero pixels in the image to be processed, which are spatial indexes of the non-zero pixels; a quantity determination module, configured to determine the number of non-zero pixel features in the image to be processed; The sequence determination module is used to determine the sequence index of the non-zero pixel points according to the number of non-zero pixel points in the image to be processed and the coordinate positions of the non-zero pixel points.

21. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

23. A computer program product, wherein The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 10 when executed by a processor.

Images