Image detection method and device based on sparse point cloud, equipment and storage medium

Abstract

The present disclosure provides a sparse point cloud-based image detection method and device, equipment and storage medium, relating to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning and the like, and can be applied to scenarios such as automatic driving and robot navigation. The specific implementation scheme is as follows: obtaining a to-be-processed image, determining the coordinate positions of non-zero points in the to-be-processed image; the to-be-processed image is sparse point cloud data, and the non-zero points represent pixel points containing pixel information in the to-be-processed image; according to the coordinate positions of the non-zero points, the offset of the pixel points in a preset size image region is determined; the offset represents the degree of offset of the pixel points out of the preset size image region, and the non-zero points correspond to the image regions one by one; according to the coordinate positions of the non-zero points and the offsets of the pixel points in the image regions, target detection is performed on the objects in the to-be-processed image. By determining the offset of the pixel points, the accuracy of image processing is improved.

Description

Technical Field

[0001] The present disclosure relates to computer vision, deep learning and other technical fields in the field of artificial intelligence technology, and can be applied to scenarios such as autonomous driving and robot navigation. In particular, it relates to an image detection method, device, equipment and storage medium based on sparse point cloud. Background Art

[0002] With the rapid development of 3D scanning technology and sensor technology, point cloud data has become an important source of information in 3D perception and reconstruction tasks.

[0003] However, point cloud data is usually sparse and irregular, which makes it challenging for traditional convolutional neural networks to process such sparse point cloud image data, resulting in low image processing accuracy and efficiency. Summary of the Invention

[0004] The present disclosure provides an image detection method, apparatus, device and storage medium based on sparse point cloud.

[0005] According to a first aspect of the present disclosure, there is provided an image detection method based on a sparse point cloud, comprising:

[0006] Acquire an image to be processed, and determine the coordinate positions of non-zero points in the image to be processed; wherein the image to be processed is sparse point cloud data, and the non-zero points represent pixel points containing pixel information in the image to be processed;

[0007] Determining an offset of a pixel point within an image region of a preset size based on the coordinate position of the non-zero point; wherein the offset represents the degree to which the pixel point is offset outside the image region of the preset size, and the non-zero point corresponds to the image region in a one-to-one manner;

[0008] Target detection is performed on the object in the image to be processed according to the coordinate position of each non-zero point and the offset of the pixel points in each image area.

[0009] According to a second aspect of the present disclosure, there is provided an image detection device based on a sparse point cloud, comprising:

[0010] An image acquisition unit, configured to acquire an image to be processed and determine the coordinate positions of non-zero points in the image to be processed; wherein the image to be processed is sparse point cloud data, and the non-zero points represent pixel points containing pixel information in the image to be processed;

[0011] an offset determination unit, configured to determine an offset of a pixel point within an image region of a preset size based on the coordinate position of the non-zero point; wherein the offset represents the degree to which the pixel point is offset outside the image region of the preset size, and the non-zero point corresponds to the image region in a one-to-one manner;

[0012] The target detection unit is used to perform target detection on the object in the image to be processed according to the coordinate position of each non-zero point and the offset of the pixel points in each image area.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively coupled to the at least one processor;

[0016] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to the first aspect of the present disclosure.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect of the present disclosure when executed by a processor.

[0019] According to the technology disclosed in the present invention, the processing accuracy of sparse point cloud data is improved, thereby improving the detection accuracy of objects in images.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0022] Figure 1 1 is a flow chart of an image detection method based on sparse point cloud provided in accordance with an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of non-zero points in an image to be processed according to an embodiment of the present disclosure;

[0024] Figure 3 1 is a flow chart of an image detection method based on sparse point cloud provided in accordance with an embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of a sequential index of non-zero points provided according to an embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram of a target point provided according to an embodiment of the present disclosure;

[0027] Figure 6 1 is a flow chart of an image detection method based on sparse point cloud provided in accordance with an embodiment of the present disclosure;

[0028] Figure 7 is a structural block diagram of an image detection device based on sparse point cloud provided according to an embodiment of the present disclosure;

[0029] Figure 8 is a structural block diagram of an image detection device based on sparse point cloud provided according to an embodiment of the present disclosure;

[0030] Figure 9 is a block diagram of an electronic device for implementing the sparse point cloud-based image detection method according to an embodiment of the present disclosure;

[0031] Figure 10 It is a block diagram of an electronic device used to implement the sparse point cloud-based image detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] With the rapid development of 3D scanning and sensor technologies, point cloud data has become an important source of information for 3D perception and reconstruction tasks. However, point cloud data is often sparse and irregular. In other words, 3D scanning often produces images with sparse point cloud data. Traditional convolutional neural networks (CNNs) face challenges processing sparse point cloud data, such as high computational complexity, high memory usage, and low convolution kernel utilization.

[0034] Sparse convolution can only perform calculations at the locations of non-empty pixels, thereby reducing the amount of calculations and improving computational efficiency. However, the convolution kernel of sparse convolution is fixed in shape, and it still has limitations when processing point cloud data with irregular structures. In other words, the current sparse convolution kernel has a fixed shape and cannot adapt to the irregular structure of point cloud data, which limits the ability to extract features and affects the accuracy and efficiency of target detection in images.

[0035] The present disclosure provides a sparse point cloud-based image detection method, apparatus, device, and storage medium, which are applied to computer vision, deep learning, and other technical fields in the field of artificial intelligence technology. They can be applied to scenarios such as autonomous driving and robot navigation to improve the accuracy of target detection in sparse point cloud data.

[0036] It should be noted that the data in this embodiment is not data for a specific user and cannot reflect the personal information of a specific user. It should be noted that the images in this embodiment are from a public data set.

[0037] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0038] In order to enable readers to have a deeper understanding of the implementation principle of this disclosure, the following Figures 1-10 The embodiment is further refined.

[0039] Figure 1 FIG. 1 is a flow chart of an image detection method based on sparse point clouds according to an embodiment of the present disclosure. The method can be performed by an image detection device based on sparse point clouds. Figure 1 As shown, the method includes the following steps:

[0040] S101 , obtaining an image to be processed, and determining the coordinate positions of non-zero points in the image to be processed; wherein the image to be processed is sparse point cloud data, and the non-zero points represent pixel points containing pixel information in the image to be processed.

[0041] Exemplarily, the image to be processed may be an image acquired by a sensor such as a lidar or an RGB-D (Red Green Blue-Depth) camera, and the image to be processed may be point cloud data. In this embodiment, the image to be processed is sparse point cloud data. The point cloud in sparse point cloud data is sparsely or irregularly distributed, so some locations in the image to be processed have pixel information while others do not. Pixels in the image to be processed that contain pixel information are determined as non-zero points, and pixels that do not contain pixel information are determined as zero points.

[0042] For non-zero points in the image to be processed, the coordinate positions of each non-zero point can be obtained. In this embodiment, the coordinate position can be a representation of 2D pixel coordinates, that is, the coordinate position can be expressed as (x, y).

[0043] Figure 2 Schematic diagram of non-zero points in the image to be processed. Figure 2 In the figure, the squares filled with diagonal lines are non-zero points, and the white squares are zero points. Figure 2There are 9 non-zero points in , which are irregularly distributed in the image to be processed.

[0044] S102. Determine the offset of the pixel points within the image area of ​​a preset size based on the coordinate position of the non-zero point; wherein the offset represents the degree to which the pixel points are offset outside the image area of ​​the preset size, and the non-zero points correspond to the image areas one-to-one.

[0045] For example, each non-zero point can correspond to an image region, where the image region is a local area in the image to be processed. Based on the coordinate position of the non-zero point, the image region corresponding to the non-zero point is determined on the image to be processed. For example, the image region for the non-zero point can be defined as a range extending outward from the non-zero point as the center point. Image regions for different non-zero points may overlap.

[0046] The size of the image region is preset, for example, the size of the preset image region is 3×3, and a 3×3 image region can be determined based on the position of the non-zero point. The non-zero point can be located in the corresponding image region.

[0047] Each image region includes multiple pixels. For example, for a 3×3 image region, the image region includes 9 pixels. For each image region, the offset of each pixel in the image region can be determined. That is, for each non-zero point, there can be a corresponding offset of multiple pixels. The offset can represent the degree to which the pixel in the image region is offset outside the image region. The offset can include the offset in the x-axis direction and the offset in the y-axis direction. For example, for the pixel in the first row and first column of the image region, the offset of the pixel can be determined to be the distance of one pixel to the left.

[0048] A preset convolution layer may be used to determine the offset of each pixel. For example, a 3×3 normal submanifold sparse convolution with a bias may be used to predict the offset of each pixel. In this embodiment, the offset determination process is not specifically limited.

[0049] S103 , performing target detection on the object in the image to be processed according to the coordinate position of each non-zero point and the offset of the pixel points in each image area.

[0050] For example, the offset of each pixel in each image region is determined, and based on the coordinate positions of all non-zero points and the offsets of all pixels in all image regions, a convolution process is performed on the image to be processed, and the feature vector of the object is extracted from the image to be processed, thereby detecting the object from the image to be processed. For example, a 3×3 convolution kernel can be preset for convolution processing. Based on the coordinate positions of the non-zero points and the offsets of the pixels in the image region corresponding to the non-zero points, the shape of the convolution kernel can be changed to move the position of the convolution kernel to the non-zero points, and feature extraction is performed on the non-zero points, thereby improving the flexibility of the convolution.

[0051] In this embodiment, a submanifold sparse convolution kernel can be used when extracting features from an image to be processed. The submanifold sparse convolution kernel has a fixed shape and cannot adapt to the irregular structure of point cloud data, which limits the ability to extract features. The size of the convolution kernel can be determined based on the preset size of the image area, and the size of the convolution kernel can be consistent with the size of the image area. In this embodiment, based on the offset of the pixel points in the image area, the pixel points at the corresponding position in the convolution kernel can be offset, so that the pixel points in the convolution kernel are offset to a non-zero point, thereby improving the accuracy and efficiency of target detection.

[0052] In the embodiment of the present disclosure, for the image to be processed of the sparse point cloud, the coordinate position of each non-zero point in the image to be processed is determined, and based on the coordinate position of the non-zero point, the image area corresponding to the non-zero point is determined. The offset of each pixel point in the image area is determined, so that the non-zero point in the image to be processed is accurately convolved according to the coordinate position of the non-zero point and the offset of each pixel point in the image area. The current convolution processing process is limited to the size range of the convolution kernel. The embodiment of the present disclosure can quickly find the non-zero point during the convolution processing through the coordinate position of the non-zero point and the offset of the pixel point in the image area, thereby adapting to the structure of irregular point cloud data and being able to quickly and accurately identify the target object from the image to be processed. By processing the non-zero points, the amount of computation for processing the point cloud data is greatly reduced, the ability of feature extraction is improved, and the accuracy and efficiency of target detection are thereby improved.

[0053] Figure 3 A flowchart of an image detection method based on sparse point clouds provided in an embodiment of the present disclosure.

[0054] In this embodiment, target detection is performed on objects in the image to be processed based on the coordinate positions of each non-zero point and the offset of the pixel points in each image area, including: determining the number of non-zero points in the image to be processed; determining the sequence index of the non-zero points based on the number of non-zero points in the image to be processed and the coordinate positions of the non-zero points; wherein the sequence index represents the sorting number of the non-zero points in the image to be processed; and target detection is performed on objects in the image to be processed based on the sequence index of each non-zero point and the offset of the pixel points in each image area.

[0055] This embodiment is based on the above embodiment. Figure 3 As shown, the method includes the following steps:

[0056] S301 , obtaining an image to be processed, and determining the coordinate positions of non-zero points in the image to be processed; wherein the image to be processed is sparse point cloud data, and the non-zero points represent pixel points containing pixel information in the image to be processed.

[0057] For example, this step may refer to the above-mentioned step S101 and will not be described in detail.

[0058] S302. Determine the offset of the pixel points within the image area of ​​a preset size based on the coordinate position of the non-zero point; wherein the offset represents the degree to which the pixel points are offset outside the image area of ​​the preset size, and the non-zero point corresponds to the image area one-to-one.

[0059] For example, this step may refer to the above-mentioned step S102 and will not be described in detail.

[0060] S303: Determine the number of non-zero points in the image to be processed.

[0061] For example, when determining the coordinate positions of the non-zero points in the image to be processed, the number of the non-zero points in the image to be processed may also be determined. For example, the number of non-zero points is 9, and each non-zero point has its own corresponding coordinate position.

[0062] S304 , determining a sequence index of the non-zero points according to the number of non-zero points and the coordinate positions of the non-zero points in the image to be processed; wherein the sequence index represents the order number of the non-zero points in the image to be processed.

[0063] For example, for each non-zero point, a sequential index of the non-zero point is determined. The sequential index represents the pixel order of the non-zero point in the image to be processed. The sequential index can be an integer starting from 0. That is, each non-zero point is numbered starting from 0. For example, if there are 6 non-zero points in the image to be processed, the non-zero points can be sorted starting from 0 according to the order of their appearance in the image to be processed. The sequential indices of the non-zero points are 0, 1, 2, 3, 4, and 5, respectively. Starting from the first row of pixels, the non-zero points can be searched for and numbered according to the order of the pixels from left to right.

[0064] In other words, a unique sequential index is assigned to each non-zero point in the image to be processed. The non-zero points can be sorted based on their number and coordinate positions, and the sorted numbers can be used as sequential indices for the non-zero points. For example, starting from the first pixel in the first row of the image to be processed, sequential indices can be assigned to each non-zero point from left to right and from top to bottom. Figure 4 Schematic diagram of the sequential index of non-zero points. Figure 4 There are 9 non-zero points in total, with sequential indexes from 0 to 8.

[0065] S305 : Perform target detection on the object in the image to be processed according to the sequential index of each non-zero point and the offset of the pixel points in each image area.

[0066] Exemplarily, the sequential indices of all non-zero points are determined, and based on the sequential indices of these non-zero points and the offsets of the pixel points in the image area corresponding to each non-zero point, the non-zero points near the convolution kernel are searched based on a preset convolution kernel, so as to perform feature extraction processing on the non-zero points near the convolution kernel and detect the object from the image to be processed.

[0067] For example, the convolution kernel can be sequentially shifted on the image to be processed based on the sequential index of the non-zero points, moving the center of the convolution kernel to the non-zero point. Based on the offset of each pixel corresponding to the non-zero point, non-zero points outside the convolution kernel range are found, and features are extracted from non-zero points outside the convolution kernel range, reducing the limitations of the convolution kernel range and achieving flexible changes in the convolution kernel shape.

[0068] In this embodiment, the position for feature extraction is adjusted according to the coordinate positions of the non-zero points and the predicted offsets, thereby better adapting to images with irregular distribution of non-zero points and improving the efficiency and accuracy of target detection.

[0069] In this embodiment, target detection is performed on objects in the image to be processed based on the sequential index of each non-zero point and the offset of the pixel points in each image area, including: determining the neighboring area of ​​the pixel point based on the offset of the pixel point in the image area corresponding to the non-zero point; wherein the neighboring area is located outside the image area corresponding to the non-zero point; if it is determined that there is a target point in the neighboring area of ​​the pixel point, determining the sequential index of the target point; wherein the target point represents other non-zero points in the image to be processed; target detection is performed on objects in the image to be processed based on the sequential index of each non-zero point and the sequential index of the target point corresponding to each non-zero point; wherein the target point corresponding to the non-zero point represents the target point corresponding to each pixel point in the image area corresponding to the non-zero point.

[0070] Specifically, for each non-zero point, the neighboring area of ​​each pixel in the image region corresponding to the non-zero point is determined based on the offset of the pixel within the image region corresponding to the non-zero point. The neighboring area is an area located outside the image region corresponding to the non-zero point and adjacent to the image region. The size of the neighboring area corresponding to different pixels can be different. The size of the neighboring area can be preset or calculated based on the offset of the pixel. For example, if the offset is an integer, the neighboring area is the size of one pixel adjacent to the pixel; if the offset is a decimal, the neighboring area is the size of two or four pixels adjacent to the pixel.

[0071] Traverse each pixel point in the adjacent area to determine whether there is a non-zero point in each adjacent area. If so, determine the non-zero point in the adjacent area as the target point. A neighboring area can have one or more target points. In this embodiment, the traversal order of the pixel points in the neighboring area is not specifically limited. If it is determined that there is a target point in the neighboring area of ​​the pixel point, determine the sequential index of the target point. It is also possible to determine whether the pixel point in the image area is a non-zero point. If the pixel point in the image area is a non-zero point, it is not necessary to determine the neighboring area of ​​the pixel point; if the pixel point in the image area is not a non-zero point, determine the neighboring area of ​​the pixel point, and search for the target point from the neighboring area. That is, the target point refers to the non-zero point in the image to be processed, excluding the non-zero point in the image area.

[0072] Figure 5 Schematic diagram of the target point. Figure 5 In the image, the image region is 3×3 in size, with two non-zero points and seven pixels corresponding to their own neighboring regions. The target point is found in the neighboring regions and its ordinal index is determined. For example, the ordinal index of the target point for the first pixel in the image region is 1, and the ordinal index of the target point for the second pixel in the image region is 0.

[0073] Each non-zero point corresponds to an image region. An image region may correspond to one or more neighboring regions, and a neighboring region may contain one or more target points. That is, a non-zero point may correspond to one or more target points. The sequential indices of the target points corresponding to the non-zero points are determined, and target detection is performed on the objects in the processed image based on the sequential indices of the non-zero points and the sequential indices of the target points corresponding to the non-zero points. For example, when performing convolution processing on the image region corresponding to the non-zero points, features of the target points corresponding to the non-zero points can be extracted based on the sequential indices of the target points corresponding to the non-zero points to obtain a feature vector corresponding to the image region.

[0074] The beneficial effect of this setting is that it can extract features of target points outside the image area, avoiding the situation where the convolution range is limited to the image area during convolution processing, effectively reducing the amount of convolution calculation and improving the accuracy and efficiency of target detection.

[0075] In this embodiment, the adjacent area of ​​the pixel point is determined based on the offset of the pixel point in the image area corresponding to the non-zero point, including: determining the offset position of the pixel point based on the offset of the pixel point in the image area corresponding to the non-zero point and the coordinate position of the pixel point in the image to be processed; determining the adjacent area of ​​the pixel point based on the offset position of the pixel point.

[0076] Specifically, the offset can represent the distance that the pixel point needs to be offset, determine the coordinate position of the pixel point in the image area, and based on the coordinate position of the pixel point and the offset corresponding to the pixel point, the coordinate position of the pixel point after the offset can be determined as the offset position. For example, the coordinate position of the pixel point can be added to the offset to obtain the offset position. The offset can include an offset in the x direction and an offset in the y direction. The x value in the coordinate position and the offset in the x direction can be added to obtain the x in the offset position, and the y value in the coordinate position and the offset in the y direction can be added to obtain the y in the offset position.

[0077] Based on the offset position of the pixel, the neighboring area of ​​the pixel is determined based on the linear interpolation algorithm. For example, if a bilinear interpolation algorithm is used, the size of the neighboring area is 4 pixels; if a trilinear interpolation algorithm is used, the size of the neighboring area is 8 pixels. For 2D pixel coordinates, using the bilinear interpolation algorithm, the position of the pixel in the neighboring area can be:

[0078]

[0079] Among them, P represents the position of the pixel in the adjacent area, p x Indicates the horizontal coordinate of the offset position, p y Indicates the vertical coordinate of the offset position. The four pixel points in the adjacent area are (px , p y )、(p x +1, p y )、(p x , p y +1)、(p x +1, p y +1), these four pixels form a neighboring area.

[0080] The beneficial effect of this setting is that the predicted offset is added to the coordinate position of the pixel point in the image area to obtain the offset position, and the area around the offset position is used as the neighboring area of ​​the pixel point, which expands the search range of the target point and improves the accuracy and efficiency of target detection.

[0081] In this embodiment, the method also includes: associating and storing the pixel index of the pixel point in the image area with the sequential index of the target point corresponding to the pixel point, and associating and storing the pixel index of the pixel point in the image area with the sequential index of the non-zero point corresponding to the image area; wherein the pixel index represents the sorting number of the pixel point in the image area.

[0082] Specifically, the number of pixels in each image region is consistent. For example, for a 3×3 image region, the number of pixels in each image region is 9. A pixel index is determined for each pixel in the image region. The pixel index represents the order of the pixel in the image region. For example, if the number of pixels is 9, the pixel index in each image region is 0 to 8. The pixel in the first row and first column of the image region can be assigned a pixel index of 0, and the pixel in the third row and third column can be assigned a pixel index of 8, in a left-to-right and then top-to-bottom order.

[0083] For each pixel in each image region, determine the sequential index of the target point corresponding to that pixel, and associate the pixel index with the sequential index of the corresponding target point. For example, for the first image region, the sequential index of the target point corresponding to the first pixel is 1; for the second image region, the sequential index of the target point corresponding to the first pixel is 0. In this case, pixel index 0 can be associated with sequential indices 1 and 0 and stored, i.e., the sequential index associated with pixel index 0 is [1, 0]. Alternatively, the pixel index of a pixel in an image region can be associated with the sequential index of the non-zero point corresponding to that image region and stored.

[0084] Two code tables are pre-built. The first is used to store the association between pixel index and the sequential index of the target point, and the second is used to store the association between pixel index and the sequential index of the non-zero point. The number of rows in these two code tables is the same, which is the number of pixels in the image area. Each row represents a pixel in the image area. In this embodiment, the number of columns in the code table is N×2 dim , N represents the number of nonzero points, and dim represents the linear interpolation algorithm used. If bilinear interpolation is used, dim is 2; if trilinear interpolation is used, dim is 3. The initial value of the code table can be set to -1. The number of sequential indices of the target points corresponding to the pixel index in the code table is the same as the number of sequential indices of the corresponding nonzero points. Sequential indices in the same row of the code table can be repeated.

[0085] For each image region, the sequential index of the non-zero point at the center of the image region can be determined first, and then the target point that affects the convolution of the non-zero point can be determined. That is, starting from the first non-zero point, the center point of the image region is placed at the position of the non-zero point, and starting from the first pixel in the image region, it is determined whether there is a target point in the adjacent area. If it is determined that there is a target point in the adjacent area of ​​the sixth pixel, the sequential index of the target point is stored in the sixth row and first column of the first code table, and the sequential index of the non-zero point at the center of the image region is stored in the sixth row and first column of the second code table. If there are two target points in the adjacent area, the sequential index of the second target point is stored in the sixth row and second column of the first code table, and the sequential index of the non-zero point at the center of the image region is repeated in the sixth row and second column of the second code table.

[0086] The beneficial effect of this setting is that two code tables are constructed, the sequential index of the target point is the index of the input feature, and the sequential index of the non-zero point is the index of the output feature, which makes it easy to find the target points that have an impact on the convolution of each non-zero point, so that when performing convolution processing, the features of the affected target points are extracted to avoid feature omission and improve the accuracy of target detection.

[0087] In this embodiment, the method also includes: if it is determined that there is a target point in the vicinity of the pixel point, then determining the interpolation coefficient of the target point based on the offset position; wherein the interpolation coefficient is used to perform a convolution operation according to the linear interpolation method; and the pixel index of the pixel point in the image area is associated with the interpolation coefficient of the corresponding target point and stored.

[0088] Specifically, since linear interpolation is used to find each non-zero target point, an interpolation coefficient needs to be introduced during the convolution calculation to characterize the influence of each target point. For each target point corresponding to each pixel point, the interpolation coefficient of the target point can be determined. The calculation formula of the interpolation coefficient can be:

[0089]

[0090] Where coeff is the interpolation coefficient in the neighboring area. There are four pixels in the neighboring area, and each pixel corresponds to an interpolation coefficient. That is, the interpolation coefficients in the first row of the above formula are the interpolation coefficients of the first pixel in the neighboring area; the interpolation coefficients in the second row of the above formula are the interpolation coefficients of the second pixel in the neighboring area; the interpolation coefficients in the third row of the above formula are the interpolation coefficients of the third pixel in the neighboring area; and the interpolation coefficients in the fourth row of the above formula are the interpolation coefficients of the fourth pixel in the neighboring area.

[0091] The pixel index of the pixel point in the image region is associated and stored with the sequential index of the corresponding target point. The interpolation coefficient of the target point is determined, and the pixel index of the pixel point in the image region is associated and stored with the interpolation coefficient of the corresponding target point. The gradient of the interpolation coefficient can also be calculated. The gradient of the interpolation coefficient is obtained by taking the derivative of the interpolation coefficient. The pixel index of the pixel point in the image region can be associated and stored with the gradient of the interpolation coefficient of the corresponding target point.

[0092] The beneficial effect of this setting is that different positions in the adjacent area correspond to different interpolation coefficients. By storing and managing the interpolation coefficients, the corresponding interpolation coefficients can be found when performing convolution calculations, thereby improving the accuracy and efficiency of convolution calculations.

[0093] In this embodiment, target detection is performed on objects in the image to be processed based on the sequential index of each non-zero point and the sequential index of the target point corresponding to each non-zero point, including: determining the input feature data of the pixel index based on the sequential index of all target points associated with the pixel index; wherein the input feature data represents the feature vector participating in the convolution operation corresponding to the pixel index; target detection is performed on objects in the image to be processed based on the input feature data of each pixel index and the sequential index of all non-zero points associated with each pixel index.

[0094] Specifically, for each pixel index, the sequential indexes of all target points associated with the pixel index are determined. The sequential indexes of all target points associated with each row of pixel indices can be determined from the code table. Based on the sequential indexes of all target points associated with the pixel index, the input feature data of the pixel index is determined. The input feature data represents the feature vector participating in the convolution operation in the adjacent area corresponding to the pixel index. For example, if the sequential index of the target point corresponding to the pixel index is [0, 2] and the target point is a non-zero point, the pixel information of the non-zero points with sequential indices 0 and 2 can be extracted from the image to be processed as the input feature data of the pixel index.

[0095] Determine the input feature data of all pixel indices. If the pixel index does not have an associated sequential index of a target point, then there is no corresponding input feature data for the pixel index. Based on the input feature data of all pixel indices and the sequential index of the non-zero points associated with these pixel indices, extract the relevant information of the object from the image to be processed, and use the feature vector obtained after extraction as the target feature vector. That is, the target feature vector can characterize the object in the image to be processed. For example, the input feature data can be weighted according to the sequential index of the non-zero points associated with the pixel index to obtain the target feature vector. Calculate the target feature vector of the image to be processed, and perform target detection on the object in the image to be processed based on the target feature vector. For example, based on the target feature vector, segment the image to be processed to obtain the object in the image to be processed, that is, complete the target detection of the image to be processed, and the detected target is the object in the image to be processed.

[0096] The beneficial effect of this setting is that sparse convolution calculation is performed according to the correlation relationship, targeted feature extraction is achieved for each pixel index, the flexibility of convolution is improved, and the accuracy of target detection is improved.

[0097] In this embodiment, the input feature data of the pixel index is determined based on the sequential index of all target points associated with the pixel index, including: determining the sparse feature data of the image to be processed; wherein the sparse feature data represents the non-zero points in the image to be processed; determining the rows to be extracted in the sparse feature data based on the sequential index of all target points associated with the pixel index; wherein the rows to be extracted represent the rows extracted from the matrix of the sparse feature data; obtaining the data of the rows to be extracted from the sparse feature data to constitute the input feature data of the pixel index.

[0098] Specifically, after obtaining the image to be processed, a convolution process can be performed on the image to be processed according to a preset convolution layer to obtain 2D sparse features of the image to be processed as sparse feature data. In this embodiment, the sparse feature data can represent non-zero points in the image to be processed, that is, the sparse feature data represents the non-zero features of the image to be processed. In this embodiment, the convolution process for obtaining the 2D sparse feature data is not specifically limited.

[0099] The sparse feature data may be an N×C matrix vector, where N is the number of non-zero features, that is, the number of non-zero points, and C is the feature dimension. The feature dimension may be the number of channels of the image feature, for example, the number of RGB channels.

[0100] A pixel index can be associated with one or more sequential indices of target points, determining the sequential indices of all target points associated with the pixel index. For example, the sequential indices of all target points associated with a pixel index can be expressed in the form of [a1, a2, a3, ...], where a1, a2, and a3 are sequential indices.

[0101] Sparse feature data can be a matrix vector with multiple rows and columns. The rows to be extracted from the sparse feature data are determined based on the sequential indices of the target points associated with the pixel indices. These rows represent the rows to be extracted from the sparse feature data matrix. For example, if the sequential indices of the target points associated with the pixel indices are [0, 1, 0], then the rows to be extracted are the first row, the second row, and the first row. In other words, the same row in the sparse feature data can be extracted multiple times.

[0102] The data of the row to be extracted is obtained from the sparse feature data, that is, the entire row of data to be extracted is obtained. Based on the data of the row to be extracted, the input feature data of the pixel index is formed. The data of the row to be extracted can be arranged from top to bottom to generate a new matrix vector as the input feature data. For example, if the rows to be extracted are the first row, the second row, and the first row, then the first row of the sparse feature data can be used as the first row of the input feature data, the second row of the sparse feature data can be used as the second row of the input feature data, and the first row of the sparse feature data can be used as the third row of the input feature data.

[0103] The beneficial effect of this setup is that for each row in the code table, the sequential index of the row is obtained, and the non-zero features of each row are determined based on the sparse feature data. This allows all non-zero points associated with the pixel index to be found during convolution calculations, thus breaking the limitations of the convolution kernel range and being able to find non-zero features outside the convolution kernel and within the neighborhood, improving feature extraction capabilities.

[0104] In this embodiment, target detection is performed on objects in the image to be processed based on the input feature data of each pixel index and the sequential index of all non-zero points associated with each pixel index, including: determining all interpolation coefficients associated with the pixel index; determining output feature data of the pixel index based on the input feature data of the pixel index, all interpolation coefficients associated with the pixel index, and preset convolution kernel weights; wherein the output feature data represents a feature vector after the convolution operation; target detection is performed on objects in the image to be processed based on the output feature data of each pixel index and the sequential index of all non-zero points associated with each pixel index.

[0105] Specifically, based on the pre-stored association relationship between pixel indices and interpolation coefficients, for each pixel index, all interpolation coefficients associated with the pixel index are determined. The number of interpolation coefficients associated with the pixel index is consistent with the number of sequential indices of the associated target points.

[0106] The convolution kernel weight is preset as a parameter of the convolution calculation. Different convolution kernel weights can be set for different pixel indexes. The output feature data of the pixel index is determined according to the input feature data of the pixel index, all interpolation coefficients associated with the pixel index, and the convolution kernel weight corresponding to the pixel index. The output feature data represents the feature vector of all target points associated with the pixel index after the convolution operation. The input feature data of the pixel index can be multiplied by the interpolation coefficient, and then multiplied by the convolution kernel weight to obtain the output feature data of the pixel index.

[0107] The order indexes of all non-zero points associated with each pixel index are determined. The target feature vector is calculated according to the output feature data of all pixel indexes and the order indexes of all non-zero points associated with all pixel indexes, so as to realize target detection on the object in the image to be processed. For example, the output feature data can be weighted and calculated according to the order indexes of the non-zero points associated with the pixel index to obtain the target feature vector.

[0108] The beneficial effect of such a setting is that the convolution calculation of the input feature data is realized by matrix multiplication of the input feature data, the interpolation coefficient, and the convolution kernel weight, thereby improving the efficiency and accuracy of target detection.

[0109] In this embodiment, the target detection on the object in the image to be processed is performed according to the output feature data of each pixel index and the order indexes of all non-zero points associated with each pixel index, which includes: for each pixel index, the output feature data of the pixel index is added to the same preset all-zero feature vector according to the order indexes of all non-zero points associated with the pixel index to obtain the target feature vector of the image to be processed; wherein the target feature vector represents the object in the image to be processed; and the object in the image to be processed is detected according to the target feature vector.

[0110] Specifically, for each pixel index, the order indexes of all non-zero points associated with the pixel index are determined. An all-zero feature vector is preset. The number of rows of the all-zero feature vector can be the number of non-zero points in the image to be processed, and the number of columns of the all-zero feature vector can be the maximum value of the number of columns in the output feature data. The output feature data of the pixel index is added to the all-zero feature vector according to the order indexes of the non-zero points associated with the pixel index. For example, the order indexes of the non-zero points associated with the pixel index are [0, 1, 2], and the first row of the output feature data is added to the first row of the all-zero feature vector, the second row of the output feature data is added to the second row of the all-zero feature vector, and the third row of the output feature data is added to the third row of the all-zero feature vector. That is, the corresponding row in the output feature data is added to the corresponding row in the all-zero feature vector according to the order index of the non-zero point.

[0111] The output feature data for all pixel indices is summed with the same all-zero feature vector. After all the output feature data are added, the all-zero feature vector becomes the target feature vector of the image to be processed. The target feature vector can represent the object to be detected in the image to be processed. Based on the target feature vector, the object in the image to be processed is detected and segmented to obtain the object in the image to be processed.

[0112] The beneficial effect of this setting is that the position where feature extraction needs to be performed is changed during convolution calculation through the sequential index of the target points and the sequential index of the non-zero points, that is, the shape and position of the convolution kernel are dynamically adjusted, which significantly improves the utilization rate of the convolution kernel and the feature extraction capability, thereby improving the accuracy and efficiency of three-dimensional target detection and segmentation.

[0113] In this embodiment, a neural network model can be preset as a target detection model for executing the image detection method of this embodiment to detect objects in an image. During the model training process, after obtaining the target feature vector, backpropagation can be performed based on the target feature vector to iteratively train the model. For example, based on the output feature data, model parameters such as the gradient of the output feature data, non-zero feature gradients, convolution kernel weight gradients, and offset gradients can be calculated. The model parameters can be updated with each training until the model update is complete.

[0114] In the embodiment of the present disclosure, for the image to be processed of the sparse point cloud, the coordinate position of each non-zero point in the image to be processed is determined, and based on the coordinate position of the non-zero point, the image area corresponding to the non-zero point is determined. The offset of each pixel point in the image area is determined, so that the non-zero point in the image to be processed is accurately convolved according to the coordinate position of the non-zero point and the offset of each pixel point in the image area. The current convolution processing process is limited to the size range of the convolution kernel. The embodiment of the present disclosure can quickly find the non-zero point during the convolution processing through the coordinate position of the non-zero point and the offset of the pixel point in the image area, thereby adapting to the structure of irregular point cloud data and being able to quickly and accurately identify the target object from the image to be processed. By processing the non-zero points, the amount of computation for processing the point cloud data is greatly reduced, the ability of feature extraction is improved, and the accuracy and efficiency of target detection are thereby improved.

[0115] Figure 6 A flowchart of an image detection method based on sparse point clouds provided in an embodiment of the present disclosure.

[0116] In this embodiment, the offset of the pixel point in the image region of the preset size is determined according to the coordinate position of the non-zero point, including: dividing an image region of a preset size in the to-be-processed image according to the coordinate position of the non-zero point; wherein the non-zero point is located at the center position of the image region; performing convolution processing on the pixel points in the image region to obtain the offset of the pixel points in the image region.

[0117] Based on the above embodiment, as shown in the following Figure 6 The method comprises the following steps:

[0118] S601, acquiring a to-be-processed image, and determining the coordinate position of a non-zero point in the to-be-processed image; wherein the to-be-processed image is sparse point cloud data, and the non-zero point represents a pixel point containing pixel information in the to-be-processed image.

[0119] Exemplarily, this step can refer to the above step S101, and will not be repeated here.

[0120] S602, dividing an image region of a preset size in the to-be-processed image according to the coordinate position of the non-zero point; wherein the non-zero point is located at the center position of the image region.

[0121] Exemplarily, the coordinate position of each non-zero point in the to-be-processed image is determined, and a convolution kernel of a preset size is set in advance, which is used for subsequent convolution calculation. The size of the convolution kernel is consistent with the size of the preset image region, for example, both are 3x3.

[0122] When performing convolution calculation, the convolution kernel of the preset size needs to be translated on the to-be-processed image, that is, the image region of the preset size needs to be translated. The translation can be performed in sequence according to the coordinate position of the non-zero point. When each translation is performed, it is necessary to ensure that the currently traversed non-zero point is located in the image region, so as to divide the image region containing the non-zero point. In this embodiment, each translation needs to have a non-zero point located at the center position of the image region, and other non-zero points can be located at other positions except the center position. For example, there are 6 non-zero points, and each translation needs to ensure that one of the non-zero points is located at the center position of the image region, and the translation is performed for 6 times to obtain 6 image regions, and each non-zero point corresponds to an image region. The translation can be performed from the first non-zero point to each non-zero point.

[0123] S603, performing convolution processing on the pixel points in the image region to obtain the offset of the pixel points in the image region.

[0124] For example, for each image region, a convolution process is performed on the pixels within the image region based on a preset convolution layer to obtain the offset of each pixel within the image region. For example, a 3×3 normal submanifold sparse convolution with a bias can be used to predict the offset of each pixel. In this embodiment, the process of determining the offset is not specifically limited.

[0125] In this embodiment, each non-zero point is sequentially used as the center position of an image region, thereby obtaining multiple image regions. Each pixel in the image region can correspond to its own offset. This allows the position of the convolution kernel to be repositioned based on the offset during subsequent convolution calculations, improving convolution flexibility, reducing restrictions on convolution shape, and improving target detection accuracy.

[0126] S604: Detect the object in the image to be processed based on the coordinate position of each non-zero point and the offset of the pixel points in each image area.

[0127] For example, this step may refer to the above-mentioned step S103 and will not be described in detail.

[0128] In the embodiment of the present disclosure, for the image to be processed of the sparse point cloud, the coordinate position of each non-zero point in the image to be processed is determined, and based on the coordinate position of the non-zero point, the image area corresponding to the non-zero point is determined. The offset of each pixel point in the image area is determined, so that the non-zero point in the image to be processed is accurately convolved according to the coordinate position of the non-zero point and the offset of each pixel point in the image area. The current convolution processing process is limited to the size range of the convolution kernel. The embodiment of the present disclosure can quickly find the non-zero point during the convolution processing through the coordinate position of the non-zero point and the offset of the pixel point in the image area, thereby adapting to the structure of irregular point cloud data and being able to quickly and accurately identify the target object from the image to be processed. By processing the non-zero points, the amount of computation for processing the point cloud data is greatly reduced, the ability of feature extraction is improved, and the accuracy and efficiency of target detection are thereby improved.

[0129] Figure 7 This is a structural block diagram of an image detection device based on sparse point cloud provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 7 The image detection device 700 based on sparse point cloud includes: an image acquisition unit 701, an offset determination unit 702 and a target detection unit 703.

[0130] An image acquisition unit 701 is configured to acquire an image to be processed and determine the coordinate positions of non-zero points in the image to be processed; wherein the image to be processed is sparse point cloud data, and the non-zero points represent pixel points containing pixel information in the image to be processed;

[0131] an offset determination unit 702 for determining an offset of a pixel point within an image region of a preset size based on the coordinate positions of the non-zero points; wherein the offset represents the degree to which the pixel point is offset outside the image region of the preset size, and the non-zero points correspond to the image regions in a one-to-one manner;

[0132] The target detection unit 703 is configured to perform target detection on the object in the image to be processed according to the coordinate positions of each non-zero point and the offset of the pixel points in each image area.

[0133] Figure 8 This is a structural block diagram of an image detection device based on sparse point cloud provided by an embodiment of the present disclosure, such as Figure 8 As shown, the image detection device 800 based on sparse point cloud includes an image acquisition unit 801, an offset determination unit 802 and a target detection unit 803, wherein the target detection unit 803 includes a quantity determination module 8031, a sequence determination module 8032 and a target detection module 8033.

[0134] A quantity determination module 8031 ​​is used to determine the number of non-zero points in the image to be processed;

[0135] The sequence determination module 8032 is configured to determine a sequence index of the non-zero points according to the number of non-zero points in the image to be processed and the coordinate positions of the non-zero points; wherein the sequence index represents the order number of the non-zero points in the image to be processed;

[0136] The target detection module 8033 is used to perform target detection on the object in the image to be processed according to the sequential index of each non-zero point and the offset of the pixel points in each image area.

[0137] In one example, the target detection module 8033 includes:

[0138] A region determination submodule, configured to determine a neighboring region of a pixel point based on an offset of the pixel point within the image region corresponding to the non-zero point; wherein the neighboring region is outside the image region corresponding to the non-zero point;

[0139] A target determination submodule is configured to determine a sequential index of a target point if it is determined that a target point exists in a neighborhood of the pixel point; wherein the target point represents other non-zero points in the image to be processed;

[0140] The image detection submodule is used to perform target detection on the object in the image to be processed based on the sequential index of each non-zero point and the sequential index of the target point corresponding to each non-zero point; wherein the target point corresponding to the non-zero point represents the target point corresponding to each pixel point in the image area corresponding to the non-zero point.

[0141] In one example, the region determination submodule is specifically used to:

[0142] Determining the offset position of the pixel point according to the offset amount of the pixel point in the image area corresponding to the non-zero point and the coordinate position of the pixel point in the image to be processed;

[0143] A neighboring area of ​​the pixel point is determined according to the offset position of the pixel point.

[0144] In one example, it also includes:

[0145] An associative storage unit is used to associate and store the pixel index of a pixel point in an image area with the sequential index of a target point corresponding to the pixel point, and to associate and store the pixel index of a pixel point in an image area with the sequential index of a non-zero point corresponding to the image area; wherein the pixel index represents the sorting number of the pixel point in the image area.

[0146] In one example, it also includes:

[0147] a coefficient determination unit, configured to determine an interpolation coefficient of the target point based on the offset position if it is determined that the target point exists in the neighborhood of the pixel point; wherein the interpolation coefficient is used to perform a convolution operation according to a linear interpolation method;

[0148] The coefficient storage unit is used to associate and store the pixel index of the pixel point in the image area with the interpolation coefficient of the corresponding target point.

[0149] In one example, the image detection submodule is specifically used to:

[0150] Determine input feature data of the pixel index according to the sequential indexes of all target points associated with the pixel index; wherein the input feature data represents a feature vector participating in the convolution operation corresponding to the pixel index;

[0151] Target detection is performed on objects in the image to be processed according to input feature data of each pixel index and sequential indices of all non-zero points associated with each pixel index.

[0152] In one example, the image detection submodule is specifically used to:

[0153] Determining sparse feature data of the image to be processed; wherein the sparse feature data represents non-zero points in the image to be processed;

[0154] Determining a row to be extracted from the sparse feature data according to the sequential indices of all target points associated with the pixel index; wherein the row to be extracted represents a row extracted from the matrix of the sparse feature data;

[0155] The data of the row to be extracted is obtained from the sparse feature data to form the input feature data of the pixel index.

[0156] In one example, the image detection submodule is specifically used to:

[0157] determining all interpolation coefficients associated with the pixel index;

[0158] Determining output feature data of the pixel index based on the input feature data of the pixel index, all interpolation coefficients associated with the pixel index, and a preset convolution kernel weight; wherein the output feature data represents a feature vector after the convolution operation;

[0159] Target detection is performed on objects in the image to be processed according to the output feature data of each pixel index and the sequential indexes of all non-zero points associated with each pixel index.

[0160] In one example, the image detection submodule is specifically used to:

[0161] For each pixel index, adding the output feature data of the pixel index to a preset all-zero feature vector according to the sequential indices of all non-zero points associated with the pixel index to obtain a target feature vector of the image to be processed; wherein the target feature vector represents an object in the image to be processed;

[0162] According to the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

[0163] In one example, the offset determining unit 802 includes:

[0164] A region division module, configured to divide the image to be processed into an image region of a preset size according to the coordinate position of the non-zero point; wherein the non-zero point is located at the center of the image region;

[0165] The offset determination module is used to perform convolution processing on the pixel points in the image area to obtain the offset of the pixel points in the image area.

[0166] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device.

[0167] Figure 9 A structural block diagram of an electronic device provided in an embodiment of the present disclosure, such as Figure 9As shown, the electronic device 900 includes at least one processor 902, and a memory 901 connected with the at least one processor 902 in communication; wherein the memory stores instructions executable by the at least one processor 902, and the instructions are executed by the at least one processor 902 to enable the at least one processor 902 to perform the sparse point cloud based image detection method of the present disclosure.

[0168] The electronic device 900 further includes a receiver 903 and a transmitter 904. The receiver 903 is configured to receive instructions and data transmitted by other devices, and the transmitter 904 is configured to transmit instructions and data to external devices.

[0169] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0170] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, which includes a computer program stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to enable the electronic device to perform the scheme provided in any of the above embodiments.

[0171] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0172] As Figure 10 shown, the device 1000 includes a computing unit 1001 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded into a random access memory (RAM) 1003 from a storage unit 1008. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0173] Various components in device 1000 are connected to I / O interface 1005, including an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0174] The computing unit 1001 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the sparse point cloud-based image detection method. For example, in some embodiments, the sparse point cloud-based image detection method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the sparse point cloud-based image detection method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the sparse point cloud-based image detection method in any other appropriate manner (for example, by means of firmware).

[0175] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0177] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0178] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0179] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0180] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0181] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0182] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An image detection method based on sparse point clouds, comprising: Acquire an image to be processed, and determine the coordinate positions of non-zero points in the image to be processed; wherein the image to be processed is sparse point cloud data, and the non-zero points represent pixel points containing pixel information in the image to be processed; determining, based on the coordinate positions of the non-zero points, an offset of pixels within an image region of a preset size; wherein the offset is obtained by performing a convolution process on the pixels within the image region, and represents the extent to which the pixels are offset outside the image region of the preset size, and the non-zero points correspond one-to-one to the image regions; Determining the number of non-zero points in the image to be processed; Determining a sequence index of the non-zero points according to the number of non-zero points in the image to be processed and the coordinate positions of the non-zero points; wherein the sequence index represents the order number of the non-zero points in the image to be processed; Determining a neighboring area of ​​the pixel point according to an offset of the pixel point in the image area corresponding to the non-zero point; wherein the neighboring area is outside the image area corresponding to the non-zero point; If it is determined that there is a target point in the neighborhood of the pixel point, determining the sequential index of the target point; wherein the target point represents other non-zero points in the image to be processed; Target detection is performed on objects in the image to be processed according to the sequential index of each non-zero point and the sequential index of the target point corresponding to each non-zero point; wherein the target point corresponding to the non-zero point represents the target point corresponding to each pixel point in the image area corresponding to the non-zero point; the sequential index of the non-zero point and the sequential index of the target point are used to dynamically adjust the convolution kernel when performing target detection.

2. The method according to claim 1, wherein The determining of the neighboring area of ​​the pixel point according to the offset of the pixel point in the image area corresponding to the non-zero point includes: Determining the offset position of the pixel point according to the offset amount of the pixel point in the image area corresponding to the non-zero point and the coordinate position of the pixel point in the image to be processed; A neighboring area of ​​the pixel point is determined according to the offset position of the pixel point.

3. The method according to claim 1 or 2, further comprising: The pixel index of a pixel point in an image area is associated with the sequential index of the target point corresponding to the pixel point and stored, and the pixel index of a pixel point in an image area is associated with the sequential index of the non-zero point corresponding to the image area and stored; wherein the pixel index represents the sorting number of the pixel point in the image area.

4. The method according to claim 3, further comprising: If it is determined that a target point exists in the vicinity of the pixel point, an interpolation coefficient of the target point is determined based on the offset position; wherein the interpolation coefficient is used to perform a convolution operation according to a linear interpolation method; The pixel index of the pixel point in the image area is associated with the interpolation coefficient of the corresponding target point and stored.

5. The method according to claim 4, wherein The method of performing target detection on the object in the image to be processed according to the sequential index of each non-zero point and the sequential index of the target point corresponding to each non-zero point includes: Determine input feature data of the pixel index according to the sequential indexes of all target points associated with the pixel index; wherein the input feature data represents a feature vector participating in the convolution operation corresponding to the pixel index; Target detection is performed on objects in the image to be processed according to input feature data of each pixel index and sequential indices of all non-zero points associated with each pixel index.

6. The method according to claim 5, wherein: The step of determining the input feature data of the pixel index according to the sequential indexes of all target points associated with the pixel index comprises: Determining sparse feature data of the image to be processed; wherein the sparse feature data represents non-zero points in the image to be processed; Determining a row to be extracted from the sparse feature data according to the sequential indices of all target points associated with the pixel index; wherein the row to be extracted represents a row extracted from the matrix of the sparse feature data; The data of the row to be extracted is obtained from the sparse feature data to form the input feature data of the pixel index.

7. The method according to claim 5 or 6, wherein: The method of performing target detection on an object in the image to be processed according to the input feature data of each pixel index and the sequential indexes of all non-zero points associated with each pixel index includes: determining all interpolation coefficients associated with the pixel index; Determining output feature data of the pixel index based on the input feature data of the pixel index, all interpolation coefficients associated with the pixel index, and a preset convolution kernel weight; wherein the output feature data represents a feature vector after the convolution operation; Target detection is performed on objects in the image to be processed according to the output feature data of each pixel index and the sequential indexes of all non-zero points associated with each pixel index.

8. The method according to claim 7, wherein: The method of performing target detection on an object in the image to be processed according to the output feature data of each pixel index and the sequential indexes of all non-zero points associated with each pixel index includes: For each pixel index, adding the output feature data of the pixel index to a preset all-zero feature vector according to the sequential indices of all non-zero points associated with the pixel index to obtain a target feature vector of the image to be processed; wherein the target feature vector represents an object in the image to be processed; According to the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

9. The method according to any one of claims 1-2, 4-6, and 8, wherein Determining the offset of the pixel points within the image area of ​​a preset size according to the coordinate position of the non-zero point includes: According to the coordinate position of the non-zero point, an image region of a preset size is divided in the image to be processed; wherein the non-zero point is located at the center of the image region; Convolution processing is performed on the pixel points in the image area to obtain the offset of the pixel points in the image area.

10. An image detection device based on sparse point cloud, comprising: An image acquisition unit, configured to acquire an image to be processed and determine the coordinate positions of non-zero points in the image to be processed; wherein the image to be processed is sparse point cloud data, and the non-zero points represent pixel points containing pixel information in the image to be processed; an offset determination unit, configured to determine an offset of a pixel point within an image region of a preset size based on the coordinate position of the non-zero point; wherein the offset is obtained by performing a convolution process on the pixel points within the image region, and represents the degree to which the pixel point is offset outside the image region of the preset size, and the non-zero point corresponds to the image region in a one-to-one manner; a target detection unit, configured to perform target detection on an object in the image to be processed based on the coordinate positions of each non-zero point and the offset of the pixel points in each image area; The target detection unit includes: A quantity determination module, configured to determine the number of non-zero points in the image to be processed; A sequence determination module is used to determine the sequence index of the non-zero points according to the number of non-zero points in the image to be processed and the coordinate positions of the non-zero points; wherein the sequence index represents the order number of the non-zero points in the image to be processed; a target detection module, configured to perform target detection on an object in the image to be processed based on a sequential index of each non-zero point and an offset of a pixel point in each image region; The target detection module includes: A region determination submodule, configured to determine a neighboring region of a pixel point based on an offset of the pixel point within the image region corresponding to the non-zero point; wherein the neighboring region is outside the image region corresponding to the non-zero point; A target determination submodule is configured to determine a sequential index of a target point if it is determined that a target point exists in a neighborhood of the pixel point; wherein the target point represents other non-zero points in the image to be processed; The image detection submodule is used to perform target detection on objects in the image to be processed based on the sequential index of each non-zero point and the sequential index of the target point corresponding to each non-zero point; wherein the target point corresponding to the non-zero point represents the target point corresponding to each pixel point in the image area corresponding to the non-zero point; the sequential index of the non-zero point and the sequential index of the target point are used to dynamically adjust the convolution kernel when performing target detection.

11. The device according to claim 10, wherein The region determination submodule is specifically configured to: Determining the offset position of the pixel point according to the offset amount of the pixel point in the image area corresponding to the non-zero point and the coordinate position of the pixel point in the image to be processed; A neighboring area of ​​the pixel point is determined according to the offset position of the pixel point.

12. The apparatus according to claim 10 or 11, further comprising: An associative storage unit is used to associate and store the pixel index of a pixel point in an image area with the sequential index of a target point corresponding to the pixel point, and to associate and store the pixel index of a pixel point in an image area with the sequential index of a non-zero point corresponding to the image area; wherein the pixel index represents the sorting number of the pixel point in the image area.

13. The apparatus according to claim 12, further comprising: a coefficient determination unit, configured to determine an interpolation coefficient of the target point based on the offset position if it is determined that the target point exists in the neighborhood of the pixel point; wherein the interpolation coefficient is used to perform a convolution operation according to a linear interpolation method; The coefficient storage unit is used to associate and store the pixel index of the pixel point in the image area with the interpolation coefficient of the corresponding target point.

14. The device according to claim 13, wherein The image detection submodule is specifically used to: Determine input feature data of the pixel index according to the sequential indexes of all target points associated with the pixel index; wherein the input feature data represents a feature vector participating in the convolution operation corresponding to the pixel index; Target detection is performed on objects in the image to be processed according to input feature data of each pixel index and sequential indices of all non-zero points associated with each pixel index.

15. The device according to claim 14, wherein The image detection submodule is specifically used to: Determining sparse feature data of the image to be processed; wherein the sparse feature data represents non-zero points in the image to be processed; Determining a row to be extracted from the sparse feature data according to the sequential indices of all target points associated with the pixel index; wherein the row to be extracted represents a row extracted from the matrix of the sparse feature data; The data of the row to be extracted is obtained from the sparse feature data to form the input feature data of the pixel index.

16. The device according to claim 14 or 15, wherein The image detection submodule is specifically used to: determining all interpolation coefficients associated with the pixel index; Determining output feature data of the pixel index based on the input feature data of the pixel index, all interpolation coefficients associated with the pixel index, and a preset convolution kernel weight; wherein the output feature data represents a feature vector after the convolution operation; Target detection is performed on objects in the image to be processed according to the output feature data of each pixel index and the sequential indexes of all non-zero points associated with each pixel index.

17. The device according to claim 16, wherein The image detection submodule is specifically used to: For each pixel index, adding the output feature data of the pixel index to a preset all-zero feature vector according to the sequential indices of all non-zero points associated with the pixel index to obtain a target feature vector of the image to be processed; wherein the target feature vector represents an object in the image to be processed; According to the target feature vector, target detection is performed on the object in the image to be processed to determine the object in the image to be processed.

18. The device according to any one of claims 10-11, 13-15, and 17, wherein: The offset determination unit includes: A region division module, configured to divide the image to be processed into an image region of a preset size according to the coordinate position of the non-zero point; wherein the non-zero point is located at the center of the image region; The offset determination module is used to perform convolution processing on the pixel points in the image area to obtain the offset of the pixel points in the image area.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

21. A computer program product, wherein The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 9 when the computer program is executed by a processor.

Images