Target detection method, computer program product, and electronic device

Abstract

The present application provides a target detection method, computer program product, and electronic device. A specific embodiment of the method includes: obtaining an image to be processed; the image to be processed includes a target detection object; performing quantization operations on the spatial information of the world coordinate system in which the target detection object is defined, using different voxel sizes; wherein the voxel size corresponding to the first quantization operation is larger than the voxel size corresponding to the second quantization operation; obtaining sparse voxels based on the voxels obtained from each quantization operation and a foreground feature probability map; wherein the foreground feature probability map is used to represent the probability that a pixel point in the image to be processed belongs to the foreground; sampling the target image features of the image to be processed based on the sparse voxels, and performing target detection based on the sampling results to obtain a target detection result. This method can achieve the effect of accelerating the detection rate in the process of pure visual 3D target detection.

Description

Technical Field

[0001] The present application relates to the field of information processing, and in particular to a target detection method, a computer program product, and an electronic device. Background Art

[0002] Object detection, also known as object extraction, is an image segmentation method based on the geometric and statistical characteristics of the target. Object detection in 3D space based on purely visual input (also known as pure vision 3D object detection) can be understood as detecting target objects in a high-dimensional 3D space using a low-dimensional 2D image.

[0003] In the related art, there are 3D target detection methods based on point cloud data, 2D image data, and multimodal data (point cloud data + 2D image data). Among them, although point cloud data has relatively rich geometric information and is more stable than other single-modal data, the cost of collecting point cloud sensors is relatively high. Therefore, in order to reduce costs, 3D target detection methods based on 2D image data are often used. However, the 3D target detection method based on 2D image data has the problem of low detection rate. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a target detection method, a computer program product, and an electronic device, so as to achieve the effect of accelerating the detection rate in the process of pure visual 3D target detection.

[0005] In the first aspect, an embodiment of the present application provides a target detection method, which includes: obtaining an image to be processed; the image to be processed includes a target detection object; performing a quantization operation on the spatial information of the world coordinate system where the defined target detection object is located with different voxel sizes; wherein the voxel size corresponding to the first quantization operation is larger than the voxel size corresponding to the second quantization operation; obtaining sparse voxels based on the voxels obtained from each quantization operation and the foreground feature probability map; wherein the foreground feature probability map is used to characterize the probability that the pixel points in the image to be processed belong to the foreground; sampling the target image features of the image to be processed based on the sparse voxels, and performing target detection based on the sampling results to obtain a target detection result. In this way, both detection rate and detection accuracy can be taken into account in the process of pure visual 3D target detection, so as to achieve higher detection accuracy with lower video memory consumption.

[0006] Optionally, the target image features are determined in advance based on the following steps: obtaining at least two images to be processed that contain the target detection object; wherein each image to be processed has a different acquisition perspective for the target detection object; performing a multi-scale feature extraction operation on each image to be processed to obtain a multi-scale feature; determining different eigenvalues ​​of the images to be processed under adjacent acquisition perspectives within the imaging overlap range, and calculating the average eigenvalue corresponding to the different eigenvalues, and obtaining the target image features of each image to be processed based on the average eigenvalue and the eigenvalue outside the imaging overlap range. In this way, target detection can be performed more easily based on multiple images to be processed under different acquisition perspectives in the process of pure visual 3D target detection, and the missing information when performing target detection based on a single image to be processed is compensated, so that the true situation of the target detection object can be more comprehensively reflected.

[0007] Optionally, sampling the target image features of the image to be processed based on the sparse voxels includes sampling the target image features based on the sparse voxels and an intrinsic and extrinsic parameter matrix of an acquisition device; the acquisition device is used to acquire the image to be processed. In this way, the target image features can be sampled more conveniently.

[0008] Optionally, the foreground feature probability map is obtained based on binary classification of the multi-scale features; and the voxels obtained according to each quantization operation and the foreground feature probability map are used to obtain sparse voxels, including: obtaining an initial voxel according to the first quantization operation, and further determining the foreground voxel probability that the initial voxel is a foreground voxel according to the foreground feature probability map; and determining the initial voxel corresponding to the foreground voxel probability that satisfies the voxel probability condition as the initial sparse voxel. In this way, the initial sparse voxels that satisfy the voxel probability condition can be determined, so that voxels that do not satisfy the voxel probability condition can be subsequently discarded, further reducing the occupied video memory capacity.

[0009] Optionally, obtaining sparse voxels based on the voxels obtained from each quantization operation and the foreground feature probability map further includes: quantizing each of the initial sparse voxels based on the voxel size of the second quantization operation to obtain the sparse voxels. This reduces the occupied video memory capacity and speeds up the detection rate to a certain extent.

[0010] Optionally, determining the foreground voxel probability that the initial voxel obtained from the first quantization operation is a foreground voxel based on the foreground feature probability map includes: projecting the initial voxel onto a two-dimensional plane according to an internal and external parameter matrix of an acquisition device, and determining the probability corresponding to each initial voxel in the foreground feature probability map based on pixel correspondence; wherein the acquisition device is used to acquire the image to be processed; and determining the probability corresponding to the initial voxel in the foreground feature probability map as the foreground voxel probability. In this way, the foreground voxel probability can be determined more conveniently, thereby facilitating the identification of the foreground voxel.

[0011] Optionally, performing target detection based on the sampling result to obtain the target detection result includes: fusing the sparse features obtained after sampling to obtain a fused feature; and performing target detection on the fused feature to obtain the target detection result. In this way, image features that can be combined with a point cloud detection algorithm can be obtained.

[0012] In a second aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which, when read and executed by a processor, execute the steps of the method provided in the first aspect above.

[0013] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method provided in the first aspect above are executed.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the method provided in the first aspect are executed.

[0015] Other features and advantages of the present application will be described in the following description and, in part, will become apparent from the description or be understood by practicing the embodiments of the present application. The objectives and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0017] Figure 1A flowchart of a target detection method provided in an embodiment of the present application;

[0018] Figure 2 A structural block diagram of a target detection device provided in an embodiment of the present application;

[0019] Figure 3 A schematic diagram of the structure of an electronic device for executing a target detection method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0021] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0022] It should be noted that, unless there is any conflict, the embodiments in this application or the technical features in the embodiments may be combined.

[0023] In recent years, significant progress has been made in AI-based research on computer vision, deep learning, machine learning, image processing, and image recognition. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems for simulating and extending human intelligence. AI is a comprehensive discipline encompassing numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, a key branch of AI, specifically enables machines to understand the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robotic navigation and positioning. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, autonomous driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, smart temperature measurement and other fields.

[0024] In the related art, there is a problem of low pure visual 3D target detection rate; in order to solve this problem, the present application provides a target detection method, a computer program product and an electronic device; further, by quantizing the spatial information of the world coordinate system including the target detection object through different voxel sizes, sparse voxels can be obtained based on the quantization results and the probability that the pixel points in the image to be processed belong to the foreground, and then the image to be processed can be sampled based on the sparse voxels. After target detection is performed on the sampled results, the target detection result can be obtained. Here, since the image features corresponding to the sampling results exist in the world coordinate system, and the image features required by the point cloud target detection algorithm also exist in the world coordinate system, the sampled image features can be applied to point cloud target detection algorithms such as sparsely embedded convolutional detection (SECOND) and point cloud target detection fast encoders (pointpillars fast encoders for object detection from point clouds, referred to as pointpillars). Specifically, the sampled image features can be combined with the detection head of the point cloud target detection algorithm to detect the target object. In this way, since the sparse voxels obtained by using different quantization results and the probability that the pixels in the image to be processed belong to the foreground are used to sample the image to be processed, the dimension of the image feature data is effectively reduced in the process of pure visual 3D target detection, the peak consumption of video memory is reduced, and the detection rate is accelerated, thereby solving the above-mentioned problem.

[0025] In some application scenarios, the above-mentioned target detection method can be applied to a terminal device, which can perform a sampling operation on the image to be processed based on sparse voxels, and can perform target detection based on the sampling result to obtain a target detection result. In other application scenarios, the above-mentioned target detection method can also be applied to a server that provides target detection services to the terminal device, which can perform a sampling operation on the image to be processed based on sparse voxels based on the image to be processed sent by the terminal device, and can send the target detection result obtained by performing target detection based on the sampling result to the terminal device. Exemplarily, the present application is written in the context of application to a server.

[0026] The defects existing in the solutions in the above-mentioned related technologies are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above-mentioned problems and the solutions proposed in the embodiments of the present invention below for the above-mentioned problems should all be the contributions made by the inventors to the present invention during the process of the invention.

[0027] Please refer to Figure 1, which shows a flow chart of a target detection method provided by an embodiment of the present application. In some application scenarios, the above target detection method can be applied to a server that provides image processing services. Figure 1 As shown, the target detection method includes the following steps 101 to 104.

[0028] Step 101: Acquire an image to be processed; the image to be processed includes a target detection object;

[0029] The target detection object may include, for example, people, animals, objects, etc. In practical application scenarios, for example, the target detection object may be photographed by a device such as a camera or a mobile phone that can actually capture the image to be processed to obtain the image to be processed.

[0030] Step 102: quantize the spatial information of the world coordinate system where the target detection object is located using different voxel sizes; wherein the voxel size corresponding to the first quantization operation is larger than the voxel size corresponding to the second quantization operation;

[0031] In some application scenarios, the server can define the spatial information of the world coordinate system where the target detection object is located. For example, the spatial information in the world coordinate system can be defined as [x min ,y min , z min , x max ,y max , z max ]=[-51.2m, -51.2m, -5m, 51.2m, 51.2m, 3m]. It should be noted that when defining the spatial information of the world coordinate system, the target detection object can be included therein.

[0032] After the server defines the spatial information of the world coordinate system where the target detection object is located, it can perform a quantization operation on the spatial information. Specifically, the spatial information can be quantized using voxel sizes from large to small. For example, the voxel size of the voxel [x, y, z] corresponding to the first quantization operation can be [0.6m, 0.6m, 0.6m] or [0.8m, 0.8m, 0.8m], etc., and the voxel size of the voxel [x, y, z] corresponding to the second quantization operation can be [0.1m, 0.1m, 0.1m] or [0.2m, 0.2m, 0.2m], etc. In some application scenarios, when the spatial information of the world coordinate system is quantized, it can be regarded as being equally divided by voxels of different voxel sizes.

[0033] Step 103 , obtaining sparse voxels based on the voxels obtained from each quantization operation and the foreground feature probability map; wherein the foreground feature probability map is used to represent the probability that a pixel point in the image to be processed belongs to the foreground.

[0034] After the server performs a quantization operation, it can obtain the voxels corresponding to this quantization operation. For example, after the server performs the first quantization operation, it can obtain voxels with a size of [0.8m, 0.8m, 0.8m]; after the server performs the second quantization operation, it can obtain voxels with a size of [0.2m, 0.2m, 0.2m].

[0035] In some application scenarios, the image to be processed may be composed of a foreground and a background. The server can detect each pixel of the image to be processed, determine the probability that the pixel belongs to the foreground, and then obtain a foreground feature probability map. In these application scenarios, for example, a 1*1 convolution kernel can be used to perform binary classification on the image to be processed to determine the probability that each pixel belongs to the foreground. Furthermore, the probability of each pixel belonging to the foreground can be arranged according to the position of the pixel in the image to be processed to obtain a foreground feature probability map, so as to orderly present the probability of each pixel belonging to the foreground through the foreground feature probability map.

[0036] In some application scenarios, the different voxels obtained from each quantization operation and the foreground feature probability map can be combined to obtain sparse voxels. In other words, the sparse voxels can be regarded as the voxels corresponding to the foreground pixels of the image to be processed.

[0037] Step 104 : sampling target image features of the image to be processed based on the sparse voxels, and performing target detection based on the sampling result to obtain a target detection result.

[0038] In some application scenarios, after the server obtains sparse voxels, it can sample target image features of the processed image based on the sparse voxels, and then perform object detection based on the sampled results to obtain target detection results. In these application scenarios, for example, a feature transformation network can be used for sampling based on the similarity between image features.

[0039] In these application scenarios, since the image features corresponding to the sampling results are located in the world coordinate system, and the image features required by the point cloud target detection algorithm are also located in the world coordinate system (both are in 3D space), the point cloud target detection algorithm can be used to perform target detection on the image features in the sampling results to obtain target detection results.

[0040] For example, for [x min ,y min , z min , x max ,y max , z max]=[-51.2m, -51.2m, -5m, 51.2m, 51.2m, 3m] spatial information, if the first quantization operation is performed on it with the size [0.8m, 0.8m, 0.8m], 128*128*10 initial voxels can be obtained. Then the second quantization operation can be performed based on the 128*128*10 initial voxels, and after sampling, image features with a dimension of 128*128*10n (where n is the sampling channel) can be obtained. At this time, 128*128*10n corresponds to the peak consumption of the video memory. When the size is directly quantized with [0.2m, 0.2m, 0.2m], 512*512*40 voxels will be obtained, and image features with a dimension of 512*512*40n can be obtained after sampling. At this time, 512*512*40n corresponds to the peak consumption of the video memory.

[0041] It can be seen that since different voxel sizes from large to small are used to quantify spatial information, and sparse voxels are obtained in combination with the foreground feature probability map, the sparse voxels are used to sample the image to be processed. This effectively reduces the peak memory capacity occupied by the image feature data. In addition, since the dimension of the image feature data is reduced, the computing speed is improved, thereby achieving the purpose of accelerating the detection rate.

[0042] In addition, the target detection object exists in the world coordinate system (that is, in 3D space), and the corresponding target image features are usually defined in the image space. In this way, the target image features need to be projected from the image coordinate system to the world coordinate system in order to obtain the image features in the world coordinate system. In this way, on the one hand, it is easy to cause a loss of detection accuracy due to projection operation errors, and on the other hand, repeated projection operations will also lead to reduced computational efficiency. In this embodiment, by directly sampling the target image features using sparse voxels in the world coordinate system, the sampling results obtained exist in the world coordinate system, thereby reducing the number of executions of the projection operation and improving the detection accuracy.

[0043] Therefore, in this embodiment, through the above steps 101 to 104, both detection rate and detection accuracy can be taken into account in the pure visual 3D target detection process, so as to achieve higher detection accuracy with lower video memory consumption.

[0044] In some optional implementations, the target image features are pre-determined based on the following steps:

[0045] Step 1: Acquire at least two images to be processed that contain the target detection object; wherein each image to be processed has a different acquisition perspective of the target detection object;

[0046] In some application scenarios, the server can obtain at least two images to be processed. The acquisition device can acquire the images to be processed from different acquisition perspectives. For example, when the target detection object is a car, six images to be processed from different perspectives can be captured by six cameras located in front of the car, right front, left front, rear, right rear, and left rear.

[0047] Step 2: performing a multi-scale feature extraction operation on each of the images to be processed to obtain multi-scale features;

[0048] In some application scenarios, the server can perform multi-scale feature extraction on each image to be processed. For example, the images to be processed from six perspectives can be input into the Feature Pyramid Network (FPN) model to implement multi-scale feature extraction, so that the multi-scale features corresponding to the six images to be processed can be obtained through FPN. Furthermore, the FPN can be, for example, 4 layers. For each image to be processed, it can obtain feature images at 4 scales. In this case, 4*6 feature images can be obtained for the 6 images to be processed.

[0049] Step 3: Determine the different eigenvalues ​​of the images to be processed under adjacent acquisition perspectives within the imaging overlap range, and calculate the average eigenvalue corresponding to the different eigenvalues, and obtain the target image feature of each image to be processed based on the average eigenvalue and the eigenvalues ​​outside the imaging overlap range.

[0050] In some application scenarios, the overlapping range of images to be processed from adjacent capture perspectives can be determined. For example, two images to be processed, captured from the front and right front capture perspectives, may both capture an image of the car's right mirror. In this case, the image of the car's right mirror can be considered the overlapping range of the images to be processed from these two adjacent capture perspectives.

[0051] Within the imaging overlap range, different image features correspond to different eigenvalues ​​in different images to be processed. The average eigenvalue corresponding to the different eigenvalues ​​can then be calculated. For example, in the image A to be processed captured at the front acquisition perspective, the eigenvalue of one of the feature points corresponding to the right mirror of the car is (u1, v1), and in the image B to be processed captured at the right front acquisition perspective, the eigenvalue of the same feature point corresponding to the right mirror of the car is (u2, v2). At this point, the average eigenvalue corresponding to the two different eigenvalues ​​can be calculated. It should be noted that the above u1, v1, u2, and v2 can be eigenvalues ​​of the images to be processed at adjacent acquisition perspectives with arbitrary values.

[0052] After determining the average eigenvalue corresponding to different eigenvalues ​​within the imaging overlap range, the eigenvalues ​​outside the imaging overlap range can be further determined. Here, the eigenvalues ​​outside the imaging overlap range can be regarded as the eigenvalues ​​corresponding to the features that exist only in one image to be processed.

[0053] After determining the average eigenvalue and the eigenvalues ​​outside the imaging overlap range, the server determines the various image features of the image to be processed, and then obtains the above-mentioned target image features.

[0054] In the related art, there are methods for performing 3D object detection based on a single image to be processed, such as Fully Convolutional One-Stage Monocular 3D Object Detection (FCOS3D) and Probabilistic and Geometric Depth (PGD) detection. However, since these methods only process a single image to be processed, there is a problem of missing information (such as depth information). Therefore, their detection results cannot fully reflect the actual situation of the target detection object (such as the actual posture).

[0055] In addition, there are pure visual 3D target detection algorithms in the related art, such as the 3D detector based on the deep self-attention transformation network (3D Object Detection from Multi-view Images via 3D-to-2DQueries, referred to as Detr3d) algorithm. Although target detection can also be performed based on the images to be processed under multiple acquisition perspectives, the feature value fusion process for the images to be processed under different acquisition perspectives is relatively complicated. In this implementation, the fusion process can be more easily implemented through steps 1 to 3 above to obtain the target image features.

[0056] In this implementation, through the above steps 1 to 3, target detection can be performed more easily based on multiple images to be processed under different acquisition perspectives during pure visual 3D target detection, and the missing information when performing target detection based on a single image to be processed is compensated, so that the actual situation of the target detection object can be reflected more comprehensively.

[0057] In some application scenarios, to reduce the dimensionality of the target image features, dimensionality reduction can be performed on the feature images at each scale. For example, convolution kernels such as 3*3 and 5*5 can be used for dimensionality reduction to further reduce the occupied video memory space.

[0058] In some optional implementations, the sampling of the target image features of the image to be processed based on the sparse voxels in the above-mentioned step 104 may include: sampling the target image features based on the sparse voxels and the internal and external parameter matrix of the acquisition device; the acquisition device is used to acquire the image to be processed.

[0059] In some application scenarios, the server can sample target image features of the image to be processed based on sparse voxels and the intrinsic and extrinsic parameter matrices of the acquisition device. The acquisition device here can include, for example, a camera, a mobile phone, or other device that can actually be used to acquire the image to be processed.

[0060] The internal and external parameter matrix of the acquisition device includes the internal parameter matrix and the external parameter matrix of the acquisition device. The internal parameter matrix K of the acquisition device can be, for example: Among them, f x 、f y Related to the focal length and pixel size of the camera, c x 、c y Related to the center of the camera imaging plane. The camera's extrinsic parameter matrix T can be, for example: Where R represents the rotation matrix and t represents the translation vector.

[0061] In some application scenarios, for example, sparse voxels can be projected onto a two-dimensional image based on the intrinsic and extrinsic parameter matrix of the acquisition device. This two-dimensional image can overlap with the image to be processed. In this way, target image features of the image to be processed can be sampled based on the projection values ​​corresponding to the two-dimensional image to obtain a sampling result.

[0062] In some optional implementations, the foreground feature probability map is obtained based on binary classification of the multi-scale features; and the above step 103 may include the following sub-steps:

[0063] Sub-step 1031 , obtaining an initial voxel according to the first quantization operation, and further determining the foreground voxel probability that the initial voxel is a foreground voxel according to the foreground feature probability map;

[0064] In some application scenarios, the voxels obtained after the first quantization operation can be regarded as the initial voxels. For example, after the first quantization operation is performed with a voxel size of [0.8m, 0.8m, 0.8m], the voxels obtained with a voxel size of [0.8m, 0.8m, 0.8m] can be regarded as the initial voxels.

[0065] Furthermore, the foreground voxel probability that the initial voxel is a foreground voxel can be determined according to the foreground feature probability map.

[0066] In some optional implementations, the sub-step 1031 may further include:

[0067] Sub-step A: projecting the initial voxels onto a two-dimensional plane map according to the intrinsic and extrinsic parameter matrix of an acquisition device, and determining the probability of each initial voxel in the foreground feature probability map based on pixel correspondence; wherein the acquisition device is used to acquire the image to be processed;

[0068] In some application scenarios, the probability that the initial voxel is a foreground voxel can be determined more conveniently using the intrinsic and extrinsic parameter matrix of the acquisition device. Here, the intrinsic and extrinsic parameter matrix of the acquisition device can be the matrix described above.

[0069] In these application scenarios, the initial voxel can be projected onto a two-dimensional plane map according to the intrinsic and extrinsic parameter matrix of the acquisition device. The two-dimensional plane map can overlap with the foreground feature probability map. Then, based on the pixel correspondence relationship between the two-dimensional plane map and the foreground feature probability map, the probability corresponding to the initial voxel in the foreground feature probability map can be determined. For example, after the initial voxel A is projected onto the two-dimensional plane map, it can correspond to pixel A'. Then, based on the pixel correspondence relationship, the pixel a corresponding to pixel A' in the foreground feature probability map can be determined. Then, the probability corresponding to pixel a can be determined as the probability corresponding to the initial voxel A in the foreground feature probability map.

[0070] Sub-step B: determining the probability corresponding to the initial voxel in the foreground feature probability map as the foreground voxel probability.

[0071] After the server determines the probability corresponding to the initial voxel in the foreground feature probability map, the server may determine the probability as the foreground voxel probability that the initial voxel is a foreground voxel.

[0072] In sub-step 1032 , the initial voxel corresponding to the foreground voxel probability that satisfies the voxel probability condition is determined as the initial sparse voxel.

[0073] After determining the foreground voxel probabilities corresponding to each initial voxel, the server can determine the initial voxels corresponding to the foreground voxel probabilities that meet the voxel probability conditions as initial sparse voxels. Here, the voxel probability conditions can, for example, include determining the first 1024 or first 512 initial voxels with larger foreground voxel probability values ​​as initial sparse voxels. In this way, the foreground voxel probabilities corresponding to the initial voxels can be arranged from largest to smallest according to their values, so as to facilitate the determination of the first 1024 or first 512 initial voxels.

[0074] In this implementation, through the above sub-steps 1031 to 1032, initial sparse voxels that meet the voxel probability condition can be determined, so that voxels that do not meet the voxel probability condition can be discarded later, further reducing the occupied video memory capacity.

[0075] In some optional implementations, the above step 103 may further include a sub-step 1033: quantizing each of the initial sparse voxels according to the voxel size of the second quantization operation to obtain the sparse voxels.

[0076] After determining the initial sparse voxel, the server can further quantize the initial sparse voxel according to the voxel size of the second quantization operation to obtain a sparse voxel. For example, the initial sparse voxel A with the size of [0.8m, 0.8m, 0.8m] can be further quantized with the voxel size of [0.2m, 0.2m, 0.2m] to obtain a sparse voxel (i.e., a voxel with the size of [0.2m, 0.2m, 0.2m]).

[0077] Here, since the point cloud data is a collection of data obtained by radar scanning, the foreground obtained by scanning the ray line can produce relatively dense point cloud data, while the background obtained by scanning the ray line produces relatively sparse point cloud data. Therefore, the image features corresponding to the background can be discarded, and only the image features corresponding to the foreground can be retained. In other words, only the foreground voxels corresponding to the foreground can be used to determine the sparse voxels.

[0078] In the related art, for [x min ,y min , z min , x max ,y max , z max ]=[-51.2m, -51.2m, -5m, 51.2m, 51.2m, 3m] spatial information. If it is directly quantized according to the voxel size of [0.2m, 0.2m, 0.2m], 512*512*40 voxels can be obtained. After sampling the image features based on these voxels, the dimension of the sampling result is 512*512*40.

[0079] In this implementation, for [x min ,y min , z min , x max ,y max , z max]=[-51.2m, -51.2m, -5m, 51.2m, 51.2m, 3m] spatial information. If the first quantization operation is performed on it with the size [0.8m, 0.8m, 0.8m], 128*128*10 initial voxels can be obtained. Then, among these 128*128*10 initial voxels, the first 1024 initial voxels with larger foreground feature probability values ​​can be determined as initial sparse voxels according to the voxel probability condition. The quantization operation is then performed on these 1024 initial sparse voxels with the size [0.2m, 0.2m, 0.2m] to obtain 1024*4*4*4 sparse voxels. Subsequently, the target image features are sampled based on the 1024*4*4*4 sparse voxels, and the dimension of the sampling result is 65536.

[0080] It can be seen that for the same spatial information and the same image to be processed, this implementation method abandons the background voxels corresponding to the background, reduces the occupied video memory capacity, and speeds up the detection rate to a certain extent.

[0081] In some optional implementations, the target detection based on the sampling result to obtain the target detection result includes: fusing the sparse features obtained after sampling to obtain a fused feature; and performing target detection on the fused feature to obtain the target detection result.

[0082] In some application scenarios, the sparse features obtained by sampling can be fused, and target detection can be performed based on the fused features to obtain target detection results. In this way, the fused features obtained correspond to foreground features in the world coordinate system, which can then be combined with the point cloud detection algorithm to obtain target detection results. Here, for example, fusion can be performed using a sparse convolution algorithm to obtain fused features. The fused features can then be detected using point cloud detection algorithms such as the second algorithm and the pointpillars algorithm to obtain target detection results.

[0083] Please refer to Figure 2 , which shows a structural block diagram of a target detection device provided by an embodiment of the present application. The target detection device can be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above Figure 1 The method embodiment corresponds to the embodiment that can be executed Figure 1 The various steps involved in the method embodiment and the specific functions of the device can be found in the description above. To avoid repetition, detailed description is appropriately omitted here.

[0084] Optionally, the target detection device includes an acquisition module, a quantization module, a determination module, and a detection module. The acquisition module is used to acquire an image to be processed; the image to be processed includes a target detection object; the quantization module is used to perform a quantization operation on the spatial information of the world coordinate system where the target detection object is defined, with different voxel sizes; the voxel size corresponding to the first quantization operation is larger than the voxel size corresponding to the second quantization operation; the determination module is used to obtain sparse voxels based on the voxels obtained from each quantization operation and the foreground feature probability map; the foreground feature probability map is used to represent the probability that a pixel point in the image to be processed belongs to the foreground; the detection module is used to sample the target image features of the image to be processed based on the sparse voxels, and perform target detection based on the sampling results to obtain a target detection result.

[0085] Optionally, the target image features are determined in advance based on the following steps: obtaining at least two images to be processed containing the target detection object; wherein each image to be processed has a different acquisition perspective of the target detection object; performing a multi-scale feature extraction operation on each image to be processed to obtain a multi-scale feature; determining different eigenvalues ​​of the images to be processed under adjacent acquisition perspectives within an imaging overlap range, and calculating the average eigenvalue corresponding to the different eigenvalues, and obtaining the target image features of each image to be processed based on the average eigenvalue and the eigenvalues ​​outside the imaging overlap range.

[0086] Optionally, the detection module is further used to: sample the target image features based on the sparse voxels and the intrinsic and extrinsic parameter matrix of an acquisition device; and the acquisition device is used to acquire the image to be processed.

[0087] Optionally, the foreground feature probability map is obtained based on a binary classification process of the multi-scale features; and the determination module is further used to: obtain an initial voxel based on the first quantization operation, and further determine the foreground voxel probability that the initial voxel is a foreground voxel based on the foreground feature probability map; and determine the initial voxel corresponding to the foreground voxel probability that satisfies the voxel probability condition as the initial sparse voxel.

[0088] Optionally, the determination module is further configured to: quantize each of the initial sparse voxels according to a voxel size of the second quantization operation to obtain the sparse voxels.

[0089] Optionally, the determination module is further used to: project the initial voxels onto a two-dimensional plane map according to the internal and external parameter matrix of the acquisition device, and determine the probability corresponding to each of the initial voxels in the foreground feature probability map based on the pixel correspondence; wherein the acquisition device is used to acquire the image to be processed; and determine the probability corresponding to the initial voxel in the foreground feature probability map as the foreground voxel probability.

[0090] Optionally, the detection module is further used to: fuse the sparse features obtained after sampling to obtain fused features; and perform target detection on the fused features to obtain the target detection results.

[0091] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0092] Please refer to Figure 3 , Figure 3 A structural diagram of an electronic device for executing a target detection method provided in an embodiment of the present application, the electronic device may include: at least one processor 301, such as a CPU, at least one communication interface 302, at least one memory 303 and at least one communication bus 304. Among them, the communication bus 304 is used to realize direct connection and communication between these components. Among them, the communication interface 302 of the device in the embodiment of the present application is used to communicate signaling or data with other node devices. The memory 303 can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 303 can optionally also be at least one storage device located away from the aforementioned processor. Computer-readable instructions are stored in the memory 303. When the computer-readable instructions are executed by the processor 301, the electronic device can execute the above-mentioned Figure 1 The method process shown.

[0093] I understand. Figure 3 The structure shown is only for illustration, and the electronic device may also include Figure 3 More or fewer components than shown, or with Figure 3 Different configurations shown. Figure 3 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0094] The embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the following can be performed: Figure 1 The method process in the illustrated method embodiment is performed by the electronic device.

[0095] An embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments. For example, the method may include: obtaining an image to be processed; the image to be processed includes a target detection object; performing quantization operations on the spatial information of the world coordinate system where the target detection object is defined, with different voxel sizes; wherein the voxel size corresponding to the first quantization operation is larger than the voxel size corresponding to the second quantization operation; sparse voxels are obtained based on the voxels obtained from each quantization operation and the foreground feature probability map; wherein the foreground feature probability map is used to represent the probability that a pixel point in the image to be processed belongs to the foreground; sampling the target image features of the image to be processed based on the sparse voxels, and performing target detection based on the sampling results to obtain a target detection result.

[0096] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0097] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0098] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0099] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0100] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A target detection method, characterized in that: include: Get the image to be processed; The image to be processed includes a target detection object; For the spatial information of the world coordinate system where the target detection object is located, quantization operations are performed on the spatial information at different voxel sizes; wherein the voxel size corresponding to the first quantization operation is larger than the voxel size corresponding to the second quantization operation; Obtaining sparse voxels based on the voxels obtained from each quantization operation and the foreground feature probability map; wherein the foreground feature probability map is used to represent the probability that a pixel point in the image to be processed belongs to the foreground; The target image features of the image to be processed are sampled based on the sparse voxels, and target detection is performed based on the sampling result to obtain a target detection result.

2. The method according to claim 1, characterized in that The target image features are determined in advance based on the following steps: Acquire at least two images to be processed that contain the target detection object; wherein each image to be processed has a different acquisition perspective of the target detection object; Performing a multi-scale feature extraction operation on each of the images to be processed to obtain multi-scale features; Determine different eigenvalues ​​of the images to be processed under adjacent acquisition perspectives within the imaging overlap range, calculate the average eigenvalue corresponding to the different eigenvalues, and obtain the target image feature of each of the images to be processed based on the average eigenvalue and the eigenvalues ​​outside the imaging overlap range.

3. The method according to claim 1 or 2, characterized in that The sampling of target image features of the image to be processed based on the sparse voxels includes: The target image features are sampled based on the sparse voxels and the internal and external parameter matrix of an acquisition device; the acquisition device is used to acquire the image to be processed.

4. The method according to claim 2, characterized in that The foreground feature probability map is obtained based on binary classification of the multi-scale features; as well as The sparse voxels are obtained according to the voxels obtained from each quantization operation and the foreground feature probability map, including: An initial voxel is obtained according to the first quantization operation, and further, according to the foreground feature probability map, a foreground voxel probability of the initial voxel being a foreground voxel is determined; The initial voxel corresponding to the foreground voxel probability that meets the voxel probability condition is determined as the initial sparse voxel.

5. The method according to claim 4, characterized in that The step of obtaining sparse voxels based on the voxels obtained from each quantization operation and the foreground feature probability map further includes: Each of the initial sparse voxels is quantized according to the voxel size of the second quantization operation to obtain the sparse voxel.

6. The method according to claim 4, characterized in that The step of obtaining an initial voxel according to the first quantization operation and further determining the foreground voxel probability that the initial voxel is a foreground voxel according to the foreground feature probability map includes: Projecting the initial voxels onto a two-dimensional plane map according to the internal and external parameter matrix of an acquisition device, and determining the corresponding probability of each initial voxel in the foreground feature probability map based on pixel correspondence; wherein the acquisition device is used to acquire the image to be processed; The probability corresponding to the initial voxel in the foreground feature probability map is determined as the foreground voxel probability.

7. The method according to any one of claims 1-2, 4-6, characterized in that: The target detection is performed based on the sampling result to obtain the target detection result, including: The sparse features obtained after sampling are fused to obtain fused features; Performing target detection on the fused features to obtain the target detection result.

8. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 7 is executed.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images