Target object detection method, system and device and storage medium

Abstract

The embodiment of the invention provides a target object detection method, system and device and a storage medium. The method comprises an optimization process, and the optimization process comprises the following steps: determining first information of a target object based on image information, and carrying out segmented quantization on the first information to obtain second information after segmented quantization; the first information comprises one or more of category, size and pose information of the target object; encoding the second information to obtain encoding information of the second information; splicing the coding information with the point cloud of the target object, inputting the spliced splicing information into a regression model, and obtaining third information of the target object based on the regression model; the third information is optimization information of the first information.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, system, device and storage medium for detecting a target object. Background Art

[0002] Target object detection can determine whether an image contains a target object based on the input upstream information, and is widely used in vehicle driving scenarios. For example, path planning and response for vehicle driving. Different driving modes may have different target object detection requirements, and therefore require upstream information of different richness. However, upstream information of different richness may disturb the target object detection process, resulting in low robustness of target object detection.

[0003] Therefore, it is necessary to provide a target object detection scheme so that the input information of the target object detection process has sparseness and stability, thereby improving the robustness of the target object detection. Summary of the invention

[0004] One or more embodiments of the present specification provide a method for detecting a target object, which includes an optimization process, wherein the optimization process includes: determining first information of the target object based on image information, segmenting and quantizing the first information to obtain second information after segmentation and quantization; the first information includes one or more of the category, size and posture information of the target object; encoding the second information to obtain encoded information of the second information; splicing the encoded information with the point cloud of the target object, and inputting the spliced ​​information into a regression model, and obtaining third information of the target object based on the regression model; the third information is optimized information of the first information.

[0005] One or more embodiments of the present specification provide a target object detection system, the target object detection system comprising: a quantization module, used to determine first information of the target object based on image information, and to quantize the first information in segments to obtain second information after segmented quantization; the first information comprises one or more of the category, size and posture information of the target object; an encoding module, used to encode the second information to obtain encoded information of the second information; a splicing module, used to splice the encoded information with the point cloud of the target object, and input the spliced ​​information into a regression model, and obtain third information of the target object based on the regression model; the third information is optimized information of the first information.

[0006] One or more embodiments of the present specification provide a device for detecting a target object, the device for detecting a target object comprising: at least one storage medium storing computer instructions; and at least one processor executing the computer instructions to implement a method for detecting a target object.

[0007] One or more embodiments of the present specification provide a computer-readable storage medium, wherein the storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes a method for detecting a target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0009] Figure 1 is an exemplary schematic diagram of an application scenario of a target object detection system according to some embodiments of this specification;

[0010] Figure 2 is an exemplary module diagram of a target object detection device according to some embodiments of this specification;

[0011] Figure 3 is an exemplary flow chart of an optimization process of first information for target object detection according to some embodiments of this specification;

[0012] Figure 4 is an exemplary schematic diagram of segmented quantized first information according to some embodiments of this specification;

[0013] Figure 5A and Figure 5B It is an exemplary schematic diagram of a plug-in method for an optimized process for target object detection according to some embodiments of this specification. DETAILED DESCRIPTION

[0014] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0015] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0016] As shown in this specification and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "comprise" and "include" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0017] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes.

[0018] Figure 1 It is an exemplary schematic diagram of the application scenario of the target object detection system shown in some embodiments of this specification. In some embodiments, the target object detection system 100 can be applied to a variety of vehicle driving scenarios, such as manned driving scenarios, unmanned driving scenarios of different levels (e.g., L4 level, L5 level). In some embodiments, the target object detection system 100 can improve the sparsity and stability of the input information of the target object detection process by implementing the method and / or process disclosed in this specification, thereby improving the robustness of the target object detection.

[0019] In some embodiments, Figure 1 As shown, the target object detection system 100 may include a processing device 110 , a vehicle 120 , a user terminal 130 , a storage device 140 , and a network 150 .

[0020] The processing device 110 can be used to process data and / or information from at least one component of the detection system 100 of the target object or an external data source (e.g., a cloud data center). For example, the processing device 110 can determine the first information of the target object based on the image information, quantize the first information in segments, and obtain the second information after segment quantization. For another example, the processing device 110 can encode the second information to obtain the encoded information of the second information. For another example, the processing device 110 can splice the encoded information with the point cloud of the target object, and input the spliced ​​spliced ​​information into a regression model, and obtain the third information of the target object based on the regression model. In some embodiments, the processing device 110 may include a central processing unit (CPU), a digital signal processor (DSP), a system on a chip (SoC), a microcontroller unit (MCU), a computer, a user console, etc. or any combination thereof. In some embodiments, the processing device 110 may include a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processing device 110 may be local or remote. In some embodiments, the processing device 110 may be implemented on a cloud platform. By way of example only, a cloud platform may include one or any combination of a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, etc.

[0021] The vehicle 120 may be a vehicle of the detection system 100 for the target object. In some embodiments, the vehicle 120 may include an automatic driving assistance vehicle, an unmanned vehicle, etc. In some embodiments, the vehicle 120 may receive instructions issued by the processing device 110 and complete the corresponding tasks according to the instructions. For example, the vehicle 120 may receive instructions such as entering the L4 level unmanned driving mode, exiting the unmanned driving mode, and automatically control the vehicle 120 to complete the corresponding operation. In some embodiments, the vehicle 120 may include a positioning component (e.g., a GPS module), a timing component (e.g., a timer), a sensor component (e.g., a motion sensor, a camera, a laser radar, etc.), a communication component (e.g., a GPS communication module, a short-range wireless communication module, etc.), etc. In some embodiments, the vehicle 120 may communicate with at least one component of the detection system 100 for the target object through the network 150. For example, the vehicle 120 may send the current driving mode of the vehicle to the processing device 110 through the network 150.

[0022] The user terminal 130 can realize the interaction between the user and the detection system 100 of the target object. In some embodiments, the user terminal 130 can be a terminal device used by the user when using the vehicle. In some embodiments, the user terminal 130 can receive the driving mode requirements input by the user. In some embodiments, the user terminal 130 can include one or any combination of a mobile device, a tablet computer, a laptop computer, other devices with input and / or output functions, etc.

[0023] The storage device 140 can be used to store data, instructions and / or any other information. For example, the storage device can store preset thresholds, parameters of the optimization process, parameters of the detection process of the target object, etc. In some embodiments, the storage device may include a random access memory (RAM), a read-only memory (ROM), a large capacity memory, a removable memory, a volatile read-write memory, etc. or any combination thereof. In some embodiments, the storage device can be integrated or included in one or more other components of the detection system 100 of the target object (e.g., the processing device 110, the vehicle 120, the user end 130).

[0024] The network 150 can facilitate the exchange of information and / or data. In some embodiments, one or more components of the detection system 100 of the target object (e.g., the processing device 110, the vehicle 120, the user end 130, the storage device 140) can send information and / or data to other components of the detection system 100 of the target object through the network 150. For example, the processing device 110 can obtain user input from the user end 130 via the network 150. For another example, the processing device 110 can obtain the driving mode of the vehicle from the vehicle 120 via the network 150. In some embodiments, the network 150 may include any one or more of a wired network or a wireless network. In some embodiments, the network 150 may include a cable network, an optical fiber network, a telecommunications network, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network (ZigBee), a near field communication (NFC), an in-device bus, an in-device line, a cable connection, etc. or any combination thereof. In some embodiments, the network connection between the components of the target object detection system 100 can adopt one of the above methods or multiple methods. In some embodiments, the network can be a point-to-point, shared, centralized, or other topological structure or a combination of multiple topological structures.

[0025] It is noteworthy that the detection system 100 of the target object is provided only for illustrative purposes and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various changes and modifications can be made according to the description of this specification. For example, the detection system 100 of the target object can also include a database, an information source, etc. For another example, the detection system 100 of the target object can be implemented on other devices to achieve similar or different functions. However, these changes and modifications will not deviate from the scope of this specification.

[0026] Figure 2is an exemplary module diagram of a target object detection device according to some embodiments of this specification. In some embodiments, the target object detection device 200 can be implemented by a processing device 110. In some embodiments, Figure 2 As shown, the target object detection device 200 may include a quantization module 210 , an encoding module 220 , a splicing module 230 and / or an insertion module 240 .

[0027] The quantization module 210 can be used to determine the first information of the target object based on the image information, and perform segmented quantization on the first information to obtain the segmented quantized second information. In some embodiments, the first information includes one or more of the category, size, and posture information of the target object. In some embodiments, the quantization module 210 can determine the expected quantization space based on the numerical characteristics of the data to be quantized. In some embodiments, the quantization module 210 can determine the quantization step size of the segmented quantization based on the expected quantization space and the numerical distribution characteristics of the data to be quantized.

[0028] The encoding module 220 may be configured to encode the second information to obtain encoding information of the second information.

[0029] The stitching module 230 can be used to stitch the coded information with the point cloud of the target object, and input the stitched information into the regression model to obtain the third information of the target object based on the regression model. In some embodiments, the third information is the optimization information of the first information. In some embodiments, the stitching module 230 can perform one or more of the following operations: based on the first information, obtain the coordinate position corresponding to the target object; obtain the point cloud of the target object from the laser point cloud according to the coordinate position corresponding to the target object for stitching; stitch the coded information of the second information with the point cloud of the target object to obtain the stitched information. In some embodiments, the stitching module 230 can filter out the point cloud related to the target object from the laser point cloud corresponding to the image information based on the coordinate position of the target object in the image information.

[0030] The plug-in module 240 can be used to connect the optimization process to the detection process of the target object. In some embodiments, the plug-in module 240 can determine whether to connect the optimization process to the detection process of the target object based on user input. In some embodiments, the plug-in module 240 can connect the optimization process to the detection process of the target object in response to the vehicle entering the automatic driving mode. In some embodiments, the plug-in module 240 can receive notification information sent by the vehicle and connect the optimization process to the detection process of the target object based on the notification information. In some embodiments, the plug-in module 240 can detect whether the vehicle enters the automatic driving mode based on a preset time interval, and in response to the vehicle entering the automatic driving mode, connect the optimization process to the detection process of the target object, and stop the timed detection. In some embodiments, the plug-in module 240 can determine the computing power of the target computing resources that can be used for target object detection based on the system computing resources and the occupied computing resources occupied by other computing processes; in response to the computing power of the target computing resources being less than a preset threshold, the optimization process is in an invalid state.

[0031] It should be noted that the above description of the target object detection device 200 and its modules is only for convenience of description and does not limit the present specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principle of the device, it is possible to arbitrarily combine the modules or form a sub-device to connect with other modules without deviating from the principle. In some embodiments, Figure 2 The modules disclosed in the specification may be different modules in a system, or one module may realize the functions of two or more modules. For example, each module may share a storage module, or each module may have its own storage module. Such variations are within the scope of protection of this specification.

[0032] Figure 3 It is an exemplary flow chart of the optimization process of the first information for target object detection according to some embodiments of this specification. In some embodiments, process 300 can be executed by processing device 110 or target object detection device 200. For example, process 300 can be stored in a storage device in the form of a program or instruction, and when processing device 110 or target object detection device 200 executes the instruction, process 300 can be implemented. The operational diagram of process 300 presented below is illustrative. In some embodiments, the process can be completed using one or more additional operations not described and / or one or more operations not discussed. In addition, Figure 3 The order in which the operations of flow 300 are illustrated and described below is not intended to be limiting.

[0033] Step 310 , determining first information of the target object based on the image information, and performing segmented quantization on the first information to obtain segmented quantized second information. Specifically, step 310 may be performed by the quantization module 210 .

[0034] The image information may be an image describing relevant information of the target object. In some embodiments, the format of the image information may include, but is not limited to, Joint Photographic Experts Group (JPEG) image format, Tagged Image File Format (TIFF) image format, Graphics Interchange Format (GIF) image format, Kodak Flash PiX (FPX) image format, Digital Imaging and Communications in Medicine (DICOM) image format, etc. In some embodiments, the image information may be a two-dimensional (2D, two-dimensional) image, or a three-dimensional (3D, three-dimensional) image. In some embodiments, the quantization module 210 may obtain image information using a sensor (e.g., a camera, a laser radar, etc.).

[0035] The first information may be information for describing features of the target object. In some embodiments, the first information may include one or more of the category, size, and position information of the target object.

[0036] The category of the target object may be a classification of the target object. As an example only, the category of the target object may include, but is not limited to, bicycles, cars, buses, children, the elderly, young people, motor vehicle signal lights, non-motor vehicle signal lights, pedestrian crossing signal lights, direction indicator signal lights, sidewalks, stop lines, turn arrows, road signs, road signs, electric poles, street lights, road separators, etc. In some embodiments, the quantization module 210 may obtain the category of the target object based on the image information through a classification model. In some embodiments, the classification model may include, but is not limited to, a Logistic regression model, a naive Bayes classification model, a KNN classification model, a neural network model, etc.

[0037] The size of the target object may be a parameter describing the size of the target object. As an example only, the size of the target object may include a maximum width W, a maximum length L, and a maximum height H of the target object.

[0038] The pose information of the target object may be a parameter describing the position and posture of the target object. In some embodiments, the pose information of the target object may include, but is not limited to, the three-dimensional coordinates (X, Y, Z) of the center point of the target object in a three-dimensional coordinate system (e.g., a three-dimensional coordinate system of a vehicle), the rotation angle of the center point of the target object relative to the vehicle (e.g., the center point of the vehicle), the two-dimensional bounding box of the target object, etc. Among them, the two-dimensional bounding box of the target object is the circumscribed rectangle of the projection of the target object on a two-dimensional plane (e.g., an XOY plane), which can be represented by the coordinate positions of the vertices of the two-dimensional bounding box, for example, (x1, y1), (x2, y2), (x3, y3), (x4, y4).

[0039] In some embodiments, the quantization module 210 can obtain the size of the target object and the position information of the target object from the image information through a feature recognition algorithm. In some embodiments, the feature recognition algorithm can include but is not limited to a combination of one or more of a histogram of directional gradients (HOG), a local binary pattern (Local Binary Pattern, LBP) algorithm, a scale invariant feature transform matching (Scale Invariant Feature Transform, SIFT) algorithm, a Haar feature (Haar-like) algorithm, a Fourier shape descriptor method (Fourier Shape Deors), a shape parameter method (Shape Factor), a finite element method (FiniteElement Method, FEM), a rotation function (Turning), etc.

[0040] The second information may be a parameter value describing the range to which the characteristics of the target object belong. In some embodiments, the second information may indicate the range to which the corresponding first information belongs. As an example only, the second information corresponding to the categories of the target object in the first information, such as "bicycle, car, bus", etc., may be the range to which it belongs, "vehicle"; the second information corresponding to the categories of the target object in the first information, such as "children, elderly, young people", etc., may be the range to which it belongs, "pedestrian"; the second information corresponding to the categories of the target object in the first information, such as "motor vehicle signal light, pedestrian crossing signal light, direction indicator signal light", etc., may be the range to which it belongs, "traffic signal light"; the second information corresponding to the categories of the target object in the first information, such as "sidewalk, turn arrow, road sign, road sign", etc., may be the range to which it belongs, "road sign"; the second information corresponding to the categories of the target object in the first information, such as "telephone pole, street light, road separator", etc., may be the range to which it belongs, "obstacles on the road". As another example, the second information corresponding to the maximum width of the target object in the first information, "W = 2.2m", may be the range to which it belongs, "2 to 3m".

[0041] Segmented quantization refers to the operation of segmenting the range of the target object feature and dividing the first information into a corresponding range (ie, the second information). In some embodiments, the quantization module 210 may segmentally quantize the first information based on the data to be quantized.

[0042] The data to be quantified may be all possible parameters corresponding to the target object feature. As an example only, the data to be quantified may include all possible categories corresponding to the category of the target object. As another example, the data to be quantified may include all possible widths corresponding to the maximum width of the target object.

[0043] In some embodiments, the quantization module 210 can perform segmented quantization on the category of the target object based on the data to be quantified by a clustering algorithm. Specifically, the abnormal data set can be a collection of driving abnormality data at abnormal locations in the sample area. Specifically, based on all possible categories corresponding to the category of the target object, the categories of the target objects with similar semantics can be clustered to obtain at least one classified data set, and each classified data set can be mapped to a classification range. Exemplarily, among all possible categories, "bicycles, cars, buses" are clustered into classified data set 1, and classified data 1 is mapped to the classification range "vehicle"; "children, elderly, young people" are clustered into classified data set 2, and classified data 2 is mapped to the classification range "pedestrian". In some embodiments, clustering can include but is not limited to a combination of one or more of the following algorithms: K-MEANS clustering algorithm, mean shift clustering algorithm, DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm, etc.

[0044] In some embodiments, the quantization module 210 may perform piecewise quantization on the numerical features of the target object (eg, the size and posture information of the target object) based on the data to be quantized by using a quantization step size.

[0045] The quantization step size may be the length of each range in the piecewise quantization. For example, 1m, 2m, etc. In some embodiments, the quantization module 210 may determine the quantization step size of the piecewise quantization based on the expected quantization space and the data to be quantized. The expected quantization space may be the range corresponding to the data to be quantized. In some embodiments, the quantization module 210 may determine the expected quantization space based on the numerical characteristics of the data to be quantized (for example, historical data, empirical data, etc.). Specifically, the quantization module 210 may determine the range within which all possible parameters corresponding to the target object features fall based on the numerical characteristics of the data to be quantized, and then the range within which all possible parameters corresponding to the target object features fall. For example, Figure 4 is an exemplary schematic diagram of the segmented quantized first information according to some embodiments of this specification, such as Figure 4 As shown, the data to be quantized includes the maximum width of the target object. The quantization module 210 can determine the historical maximum width of the target object in the range of 1.2 to 5.2 m based on historical data, and determine that all possible widths corresponding to the maximum width of the target object are in the range of 0 to 6 m, then the corresponding expected quantization space is 0 to 6 m. In some embodiments, the quantization module 210 can determine the quantization step of the segmented quantization based on the distribution characteristics of the data to be quantized in the expected quantization space. As an example only, the quantization step of the maximum width is determined based on the variance of the distribution of all possible widths of the target object in the expected quantization space. For example, the larger the variance, the greater the discrete degree of the maximum width, and the smaller the corresponding quantization step. Exemplarily, based on the variance of the distribution of all possible widths of the target object in the expected quantization space of 0 to 6 m, the corresponding quantization step is determined to be 1 m.

[0046] Specifically, the quantization module 210 may divide the desired quantization space into multiple ranges with a length of the quantization step, and divide the first information into one of the ranges. Figure 4 As shown, the expected quantization space 0~6m corresponding to the maximum width of the target object is divided into 6 ranges with a quantization step length of 1m: "0~1m", "1~2m", "2~3m", "3~4m", "4~5m", and "5~6m", and the maximum width of the target object "W=2.2m" is divided into "2~3m" (i.e., the second information).

[0047] In some embodiments, the second information may be represented by discrete values. As an example, the second information corresponding to the target object categories "vehicle", "pedestrian", "traffic light", "road sign", and "road obstacle" may be represented by integers of 1 to 5, respectively. Figure 4 As shown, the second information "0-1m", "1-2m", "2-3m", "3-4m", "4-5m", "5-6m" corresponding to the maximum width of the target object can be represented by integers of 1 to 6. For example, the first information can include the category of the target object "bicycle" and the maximum width of the target object "W=2.2m", and the second information can be represented by discrete values ​​1 and 3.

[0048] In some embodiments, the second information may be represented by an information vector, wherein the elements of the information vector may be discrete values ​​corresponding to the first information. For example, continuing the above example, the second information may be represented by an information vector (1, 3).

[0049] In some embodiments, the second information can be represented by a one-hot encoding. The one-hot encoding is an M-dimensional vector consisting of 1 "1" and M-1 "0" as elements, where the element "1" represents a valid bit, corresponding to one of the M ranges corresponding to the second information. It can be understood that each M-dimensional one-hot encoding can represent a range corresponding to each second information. Just as an example, continuing with the previous example, the category of the target object is divided into 5 ranges, and the second information corresponding to the first range "vehicle" to which "car" belongs can be represented by a 5-dimensional one-hot encoding (1, 0, 0, 0, 0). As another example, Figure 4 As shown, the maximum width of the target object is divided into 6 ranges, and the second information corresponding to the third range "2-3m" to which "W=2.2m" belongs can be represented by a 6-dimensional unique hot encoding (0, 1, 0, 0, 0, 0).

[0050] In some embodiments of the present specification, determining the quantization step of the segmented quantization based on the expected quantization space and the data to be quantized can ensure that the second information after the segmented quantization can accurately represent the first information, while ensuring that the second information saves storage space and avoids data redundancy.

[0051] Step 320 , encode the second information to obtain encoding information of the second information. Specifically, step 320 may be performed by the encoding module 220 .

[0052] The encoding information of the second information may be sparse information of the second information. In some embodiments, the encoding module 220 may encode the second information by an encoder and / or an encoding model. In some embodiments, the encoder and / or encoding model include, but are not limited to, a combination of one or more of a U-Net encoder, a Transformers bidirectional encoder, a BERT (Bidirectional Encoder Representations from Transformers) model, an ELMo (Embedding from Language Models) model, a CLIP (Contrastive Language-Image Pre-Training) model, and the like.

[0053] It can be understood that, compared with the first information, the encoded information of the second information becomes more sparse and has higher stability.

[0054] Step 330 , splicing the coded information with the point cloud of the target object, and inputting the spliced ​​information into the regression model, and obtaining the third information of the target object based on the regression model. Specifically, step 330 may be performed by the splicing module 230 .

[0055] The point cloud detector receives point cloud data collected by 3D sensors (such as lidar or depth cameras) and identifies key information such as posture and size.

[0056] A point cloud can be a set of points used to describe a three-dimensional space. A point cloud of a target object can be a set of points in a three-dimensional space that represent the size and position information of the target object. The points in the point cloud of a target object can correspond to voxels of the target object, including the position information and features (e.g., color features, etc.) of the corresponding voxels in the three-dimensional space.

[0057] In some embodiments, the stitching module 230 can obtain the coordinate position corresponding to the target object based on the first information. The coordinate position corresponding to the target object can be the coordinate position of the outer contour of the target object in three-dimensional space, which can be represented by the coordinate position of the vertices of the outer contour of the target object. For example, if the target object is a cuboid, the coordinate position corresponding to the target object can be represented by the coordinate position of the 8 vertices of the cuboid. As an example only, the position coordinates corresponding to the target object can be determined based on the center point coordinates of the target object, the size of the target object, the rotation angle of the center point of the target object relative to the vehicle (e.g., the center point of the vehicle), and the two-dimensional bounding box of the target object.

[0058] In some embodiments, the stitching module 230 can obtain the point cloud of the target object from the laser point cloud according to the coordinate position corresponding to the target object. The laser point cloud can be a point cloud of the three-dimensional space of the vehicle obtained by the vehicle's 3D sensor (e.g., a laser radar or a depth camera). In some embodiments, the stitching module 230 can filter out the point cloud related to the target object from the laser point cloud corresponding to the image information based on the coordinate position of the target object in the image information. As an example only, the stitching module 230 can input the coordinate position of the target object and the laser point cloud into a point cloud detector, and the point cloud detector can obtain the point corresponding to the coordinate position from the laser point cloud as the point cloud of the target object.

[0059] The stitching information may be the fusion information of the coding information and the point cloud. The stitching information may be based on the point cloud and include more shallow information on the basis of the coding information. In some embodiments, the stitching module 230 may stitch the coding information of the second information and the point cloud based on the position relationship to obtain the stitching information.

[0060] In some embodiments of the present specification, the point cloud and the encoding information of the target object are spliced ​​to obtain the spliced ​​information, so that the spliced ​​information can be integrated with more accurate and reliable size information, position information, etc.

[0061] The third information is the optimization information of the first information. Specifically, the splicing module 230 can input the splicing information into a regression model, and the regression model performs multi-layer feature extraction on the fusion information of the point cloud and the encoding information in the splicing information, and uses the extracted features as the third information. In some embodiments, the regression model can include but is not limited to a combination of one or more of a Pillar Feature Net model, a Point Net model, a Transformer model, etc. For example, taking the Point Net model as an example, the stitching module 230 can obtain the coordinate positions of the outer contour vertices of the "car" in three-dimensional space based on the first information of the "car": the size of the "car" (W=2.2, L=5, H=0.8), the center point coordinates of the "car" (0.5, 0.3, 0.6), and the two-dimensional bounding box of the "car" (x=1, y=2): D1(x1', y1', z1'), D2(x2', y2', z2'), ..., Dn(xn', yn', zn'), and obtain the point cloud of the "car" from the laser point cloud according to the coordinate positions of the n vertices corresponding to the "car", and splice the point cloud of the "car" and the encoding information of the "car" and input them into the Point Net model. The Point Net model outputs the optimized third information corresponding to the "car" based on the stitching information, that is, more accurate posture information and size information of the "car".

[0062] In some embodiments, the stitching module 230 can train a regression model based on a large number of training samples with labels. Specifically, the training samples with labels are input into the regression model, and the parameters of the regression model are updated through training. In some embodiments, the training samples may include sample coding information and sample point clouds corresponding to the target object. In some embodiments, the sample coding information and sample point clouds corresponding to the target object can be obtained based on the sample image information. For a detailed description of obtaining the sample coding information and sample point clouds corresponding to the target object based on the sample image information, please refer to the relevant description of obtaining the coding information and point clouds corresponding to the target object based on the image information in steps 310 and 320. In some embodiments, the identification can be the accurate posture information and accurate size information of the target object in the manually annotated sample image information.

[0063] It should be noted that the above description of process 300 is provided for illustrative purposes only and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various changes and modifications can be made according to the description of this specification. However, these changes and modifications do not depart from the scope of this specification. In some embodiments, process 300 may include one or more additional operations, or one or more operations may be omitted.

[0064] In some embodiments of the present specification, the first information is segmented, quantized and encoded so that the obtained encoded information is sparse valid information, thereby ensuring that the third information obtained based on the encoded information is not affected by fluctuations in the upstream first information, thereby reducing sensitivity to the first information and improving the stability of the third information and its robustness for subsequent processing.

[0065] For example, taking the maximum width of the target object as the first information, the first information can be the maximum width of the target object "Car A" W = 2.2m, the maximum width of "Bicycle B" W = 0.25m, the maximum width of "Bicycle C" W = 1.2m, the maximum width of "Bus D" W = 5.5m, etc., that is, the maximum width of the target object can be any value in the range of 0 to 6m, and the maximum width has infinite values. When the first information changes from the maximum width of "Bicycle B" W = 0.25m to the maximum width of "Bus D" W = 5.5m, the first information is equivalent to becoming 22 times the original. In addition, when the first information continues to change, the corresponding multiple can also be any multiple. If the third information is directly obtained based on the first information and the point cloud, the third information will be unstable due to the fluctuation of the first information; the segmented quantized second information can correspond to include The maximum width W = 0.5m of "Bicycle B" corresponds to a discrete value 1 or a one-hot encoding (1, 0, 0, 0, 0, 0) and the maximum width W = 5m of "Bus D" corresponds to a discrete value 6 or a one-hot encoding (0, 0, 0, 0, 0, 1). When the first information changes from the maximum width W = 0.25m of "Bicycle B" to the maximum width W = 5.5m of "Bus D", the corresponding second information changes from the discrete value 1 to 6, which is equivalent to becoming 6 times the original, or from the one-hot encoding (1, 0, 0, 0, 0, 0) to (0, 0, 0, 0, 0, 1). In addition, the second information will only be one of the 6 discrete values ​​or 6 one-hot encodings, which has sparsity and stability, so that the third information obtained based on the second information and the point cloud is also correspondingly stable. The stable third information correspondingly improves the robustness of subsequent target object detection.

[0066] Figure 5A and Figure 5B is an exemplary schematic diagram of a plug-in method for an optimization process for target object detection according to some embodiments of this specification. In some embodiments, Figure 5A and Figure 5B It can be performed by the processing device 110 or the target object detection device 200 (for example, the plug-in module 240). For example, Figure 5A and Figure 5B It can be stored in the storage device in the form of a program or instruction. When the target object detection device 200 (for example, the processing device 110) or the target object detection device 200 (for example, the plug-in module 240) executes the instruction, it can be realized. Figure 5Aand Figure 5B The following is presented Figure 5A and Figure 5B The operation diagram is illustrative. In some embodiments, the process can be completed using one or more additional operations not described and / or one or more operations not discussed. In addition, Figure 5A and Figure 5B The order in which the operations of the processes illustrated and described below are not intended to be limiting.

[0067] The detection of the target object is a process of determining whether the image information contains the target object based on the input upstream information. In some embodiments, the target object detection can be implemented based on the target detection model. In some embodiments, the target detection model can include but is not limited to a Visual Geometry Group Network model, an Inception NET model, a Fully Convolutional Network model, a Segmentation Network model, and a Mask-Region Convolutional Neural Network model.

[0068] Different target object detection requirements can correspond to different target detection accuracy. For example, the target detection accuracy applied to L0 level manned driving is low, and the target detection accuracy applied to L4 level autonomous driving is high. In order to save computing resources and meet different target object detection requirements at the same time, different upstream information can be input into the target object detection process. For example, when the target object detection requirement is for L0 level manned driving, the upstream information of the target object detection process can be the first information obtained based on the image information obtained by the sensor; when the target object detection requirement is for L4 level autonomous driving, the upstream information of the target object detection process can be the third information that integrates the image information and the point cloud.

[0069] The optimization process is a process of optimizing the first information to obtain the third information. For a detailed description of the optimization process, see Figure 3 In some embodiments, the plug-in module 240 can plug and unplug the optimization process into the detection process of the target object, so as to input different upstream information into the detection process of the target object based on different detection requirements of the target object.

[0070] In some embodiments, the plug-in module 240 can determine whether to connect the optimization process to the detection process of the target object based on user input. Specifically, the plug-in module 240 can receive the target object detection requirements input by the user through voice, text, or gestures, and determine the corresponding target object detection accuracy based on the target object detection requirements, and determine whether to connect the optimization process to the detection process of the target object based on the target object detection accuracy. For example, after the plug-in module 240 receives the user's voice input "enter L4 autonomous driving mode", it determines that the corresponding target object detection accuracy is a higher accuracy based on "entering L4 autonomous driving", and connects the optimization process to the detection process of the target object based on the target object detection accuracy being a higher accuracy.

[0071] In some embodiments of the present specification, whether to connect the optimization process to the target object detection process is determined based on user input, and the accuracy of target object detection can be determined according to user needs, thereby improving user experience.

[0072] In some embodiments, the plug-in module 240 can connect the optimization process to the detection process of the target object in response to the vehicle entering the automatic driving mode. As an example only, when the vehicle enters the automatic driving mode, a notification message can be sent to the plug-in module 240, and the plug-in module 240 receives the notification message sent by the vehicle and connects the optimization process to the detection process of the target object based on the notification message. As another example, the plug-in module 240 can detect whether the vehicle enters the automatic driving mode based on a preset interval time (for example, 5s, 10s, 20s, 30s, etc.), and when it is detected that the vehicle enters the automatic driving mode, the optimization process is connected to the detection process of the target object and the timed detection is stopped.

[0073] In some embodiments of the present specification, whether to connect the optimization process to the detection process of the target object is automatically determined based on the driving mode of the vehicle, so that the detection requirements of different target objects in different driving modes can be automatically matched, thereby improving driving physical examination.

[0074] like Figure 5A As shown, when the optimization process is connected to the detection process of the target object, the plug-in module 240 can input the first information into the optimization process, and input the third information output by the optimization process as upstream information into the detection process of the target object, so that the detection process of the target object can output more accurate detection results of the target object based on the third information containing more information.

[0075] In some embodiments, the plug-in module 240 unplugs the optimization process connected to the detection process of the target object or keeps the detection process not involved in the target object, so that the optimization process is in an invalid state.

[0076] In some embodiments, the plug-in module 240 can determine whether the optimization process is in an invalid state based on user input. For a detailed description of determining whether the optimization process is in an invalid state based on user input, please refer to the relevant description of determining whether to connect the optimization process to the target object detection process based on user input. For example, after the plug-in module 240 receives the text "Exit autonomous driving, enter manned driving" input by the user, it determines that the corresponding target object detection accuracy is a lower accuracy based on "Exit autonomous driving, enter manned driving", and unplugs the optimization process from the target object detection process based on the target object detection accuracy being a lower accuracy.

[0077] In some embodiments, the plug module 240 may be responsive to the vehicle exiting the automatic driving mode or maintaining the manned driving mode, rendering the optimization process in an invalid state. For a detailed description of the plug module 240 determining whether the vehicle exits the automatic driving mode or maintains the manned driving mode, see the description of determining whether the vehicle enters the automatic driving mode.

[0078] In some embodiments, the plug-in module 240 may be in response to the computing power of the computing resource being less than a preset threshold value, and the optimization process is in a failed state. The computing power of the computing resource can be used to evaluate the computing capacity of the computing resource. Specifically, the plug-in module 240 can determine the computing power of the computing resources that can be used for target object detection based on the system computing resources and the occupied computing resources occupied by other computing processes. The preset threshold can be the computing power of the minimum computing resource that can be used to execute the optimization process. For example, in response to the computing power "0.1TH / s" of the computing resource used for target object detection being less than the preset threshold "0.2TH / s", the optimization process is in a failed state.

[0079] In some embodiments of this specification, determining whether the optimization process is in an invalid state based on the computing power of computing resources can, on the one hand, avoid the detection process of the target object being stuck due to insufficient computing power of computing resources, thereby reducing the risk in the autonomous driving process. On the other hand, the optimization process and the detection process of the target object can be adapted to different hardware platforms to improve the applicability of the system.

[0080] like Figure 5B As shown, when the optimization process unplugs the detection process of the target object, the plug-in module 240 can directly input the first information into the detection process of the target object as upstream information, so that the detection process of the target object can directly output the detection result of the target object based on the first information.

[0081] In some embodiments of the present specification, the optimization process and the detection process of the target object are connected in series, and based on different detection requirements of the target object and / or the computing power of the computing resources, the optimization process is flexibly inserted or unplugged into the detection process of the target object. According to the connection or unplugging of the optimization process, the first information or the third information is input into the detection process of the target object. The third information sparses the dense first information, so that the third information has a stability that is not affected by the first information, and can contain rich upstream information, thereby improving the coupling between the optimization process and the detection process of the target object and realizing the pluggability of the optimization process.

[0082] It should be noted that Figure 5A and Figure 5B The above description is provided for illustrative purposes only and is not intended to limit the scope of this specification. For those of ordinary skill in the art, various changes and modifications can be made based on the description of this specification. However, these changes and modifications do not depart from the scope of this specification. In some embodiments, Figure 5A and Figure 5B One or more additional operations may be included, or one or more operations described above may be omitted.

[0083] The beneficial effects that may be brought about by the embodiments of the present specification include but are not limited to: (1) performing segmented quantization and encoding on the first information so that the obtained encoded information is sparse effective information, so that the third information obtained based on the encoded information is not affected by the fluctuation of the upstream first information, the sensitivity to the first information is reduced, and the stability of the third information and the robustness for subsequent processing are improved; (2) determining the quantization step size of the segmented quantization based on the expected quantization space and the data to be quantized can ensure that the second information after the segmented quantization can accurately represent the first information, and at the same time ensure that the second information saves storage space and avoids data redundancy; (3) connecting the optimization process and the detection process of the target object in series, and based on different detection requirements of the target object and / or the computing power of the computing resources, flexibly inserting or unplugging the optimization process into or out of the detection process of the target object, according to the insertion or unplugging of the optimization process, inputting the first information or the third information into the detection process of the target object, and the third information sparsely sparses the dense first information, so that the third information has a stability that is not affected by the first information, and can contain rich upstream information, thereby improving the coupling between the optimization process and the detection process of the target object and realizing the pluggability of the optimization process.

[0084] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other beneficial effects that may be obtained.

[0085] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to this specification. Such modifications, improvements and corrections are suggested in this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0086] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0087] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0088] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0089] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of this specification are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

[0090] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification are hereby incorporated by reference in their entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this specification (currently or later attached to this specification) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0091] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A method for detecting a target object, characterized in that: The method includes an optimization process, and the optimization process includes: Determine first information of the target object based on the image information, and perform segmented quantization on the first information to obtain segmented quantized second information; the first information includes one or more of the category, size, and posture information of the target object; Encoding the second information to obtain encoded information of the second information; The coded information is spliced ​​with the point cloud of the target object, and the spliced ​​information is input into a regression model, and third information of the target object is obtained based on the regression model; the third information is optimized information of the first information.

2. The method according to claim 1, characterized in that The determining first information of the target object based on the image information, and performing segmented quantization on the first information to obtain segmented quantized second information includes: Determining the expected quantization space based on the numerical characteristics of the data to be quantized; The quantization step size of the segmented quantization is determined based on the expected quantization space and the numerical distribution characteristics of the data to be quantized.

3. The method according to claim 1, characterized in that The step of splicing the coded information with the point cloud of the target object includes: Based on the first information, obtaining a coordinate position corresponding to the target object; Acquire a point cloud of the target object from the laser point cloud according to the coordinate position corresponding to the target object; splicing the encoded information of the second information with the point cloud of the target object to obtain the spliced ​​information; The step of obtaining the point cloud of the target object from the laser point cloud according to the coordinate position corresponding to the target object specifically includes: based on the coordinate position of the target object in the image information, filtering out the point cloud related to the target object from the laser point cloud corresponding to the image information.

4. The method according to claim 1, characterized in that Also includes: Determining whether to connect the optimization process to the detection process of the target object based on the user input includes: Receive the user's detection requirements for the target object; Determine the corresponding target object detection accuracy based on the detection requirement; Whether to connect the optimization process to the detection process of the target object is determined according to the detection accuracy of the target object.

5. The method according to claim 1, characterized in that Also includes: In response to the vehicle entering the automatic driving mode, the optimization process is connected to the detection process of the target object, including: Receiving notification information sent by the vehicle, and connecting the optimization process to the detection process of the target object based on the notification information; or Based on a preset time interval, it is detected whether the vehicle enters the automatic driving mode. In response to the vehicle entering the automatic driving mode, the optimization process is connected to the detection process of the target object, and the timed detection is stopped.

6. The method according to claim 1, characterized in that Also includes: Determining the computing power of target computing resources that can be used for detecting the target object based on system computing resources and occupied computing resources occupied by other computing processes; In response to the computing power of the target computing resource being less than a preset threshold, the optimization process is in an invalid state.

7. A target object detection system, characterized in that: The system comprises: A quantization module, configured to determine first information of a target object based on image information, and perform segmented quantization on the first information to obtain segmented quantized second information; the first information includes one or more of category, size, and posture information of the target object; an encoding module, used for encoding the second information to obtain encoding information of the second information; A splicing module is used to splice the coded information with the point cloud of the target object, and input the spliced ​​information into a regression model to obtain third information of the target object based on the regression model; the third information is the optimized information of the first information.

8. The system of claim 1, wherein: It also includes a plug-in module for: in response to the vehicle entering the automatic driving mode, connecting the optimization process to the detection process of the target object, including: Receive the user's detection requirements for the target object; Determine the corresponding target object detection accuracy based on the detection requirement; Whether to connect the optimization process to the detection process of the target object is determined according to the detection accuracy of the target object.

9. A device for detecting a target object, characterized in that: The device comprises: at least one storage medium storing computer instructions; At least one processor executes the computer instructions to implement the target object detection method according to any one of claims 1 to 6. 10 . A computer-readable storage medium storing computer instructions. When a computer reads the computer instructions, the computer executes the target object detection method according to claim 1 .

Images