Data acquisition method and device and electronic equipment

Abstract

The invention discloses a data acquisition method and device and electronic equipment, and the electronic equipment obtains three-dimensional point cloud data and a two-dimensional image which are synchronously collected from a target scene, carries out the semantic segmentation of the two-dimensional image, obtains the semantic segmentation image of the two-dimensional image, and carries out the segmentation of the two-dimensional image according to the depth information of each first collection point in a three-dimensional point cloud data set. And converting the semantic segmentation image into false point cloud data for the target scene, thereby supplementing the three-dimensional point cloud data of the corresponding target segmentation object by using the false point cloud data for the target segmentation object in the target scene, obtaining target point cloud data for the target scene, and realizing target identification of the target scene.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data acquisition method, device and electronic device. Background Art

[0002] At present, multi-sensor fusion technology is usually used in target recognition applications such as automatic / assisted driving, environmental perception, rescue services, and vehicle management. Data from multiple sensors or information sources such as lidar and cameras are integrated to compensate for their differences, thereby improving the accuracy and robustness of perception.

[0003] However, in the process of fusing the two-dimensional image obtained by the camera with the three-dimensional point cloud data obtained by the lidar, interference features extracted from the two-dimensional image (such as background environment features or other non-target object features, etc.) are often introduced into the three-dimensional point cloud data, resulting in the presence of more interfering point clouds in the fused high-density point cloud data, thereby reducing the accuracy of subsequent target recognition based on this. Summary of the invention

[0004] In view of the above problems, this application provides the following solutions:

[0005] The first aspect of the present application provides a data acquisition method, the method comprising:

[0006] Acquire the three-dimensional point cloud data and two-dimensional images collected synchronously for the target scene;

[0007] Performing semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image;

[0008] Converting the semantic segmentation image into pseudo point cloud data for the target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data set;

[0009] Using the pseudo point cloud data for the target segmented object in the target scene, the three-dimensional point cloud data corresponding to the target segmented object is supplemented to obtain target point cloud data for the target scene;

[0010] The target point cloud data is used to perform target recognition on the target scene.

[0011] In a possible implementation, the semantic segmentation image includes each segmentation object having semantic information; and using the pseudo point cloud data for the target segmentation object in the target scene to supplement the three-dimensional point cloud data corresponding to the corresponding target segmentation object to obtain the target point cloud data for the target scene includes:

[0012] Determining the pseudo point cloud data of at least one target segmentation object according to the semantic information contained in the pseudo point cloud data for the target scene;

[0013] Sampling the supplementary point cloud data corresponding to the target segmentation object from the pseudo point cloud data of each of the at least one target segmentation object;

[0014] Using the supplementary point cloud data of the at least one target segmentation object, interpolate and supplement the three-dimensional point cloud data corresponding to the target segmentation object to obtain supplemented three-dimensional point cloud data for the target scene;

[0015] The spatial position information and semantic information of each of the first acquisition points in the supplemented three-dimensional point cloud data and each of the second acquisition points supplemented by the difference constitute the target point cloud data for the target scene;

[0016] The semantic information of each of the first acquisition points is obtained based on the semantic information mapping contained in the semantic segmentation image.

[0017] In a possible implementation, the sampling of the supplementary point cloud data of the corresponding target segmentation object from the pseudo point cloud data of each of the at least one target segmentation object includes:

[0018] Determine a non-edge area of ​​each of the at least one target segmentation object; the non-edge area refers to an area where the accuracy of depth information mapped by each pixel point of the corresponding target segmentation object is greater than or equal to a threshold;

[0019] The supplementary point cloud data corresponding to the target segmentation object is sampled from the pseudo point cloud data located in the respective non-edge area of ​​the at least one target segmentation object.

[0020] In a possible implementation, determining the non-edge area of ​​each of the at least one target segmentation objects includes:

[0021] Determine the number of pixels of the same target segmentation object contained in each pixel row or pixel column in the pseudo point cloud data;

[0022] Determine edge position information of the corresponding target segmentation object in the corresponding pixel row or pixel column according to the determined number of pixels of each target segmentation object and the edge area width of the corresponding target segmentation object;

[0023] The non-edge area of ​​the corresponding target segmentation object is determined by using the edge position information of each target segmentation object in different pixel rows or pixel columns.

[0024] In a possible implementation, determining the edge position information of the corresponding target segmentation object in the corresponding pixel row or pixel column according to the determined number of pixels of each target segmentation object and the edge area width of the corresponding target segmentation object includes at least one of the following:

[0025] If the number of pixels of any of the pixel rows of the target segmentation object is greater than or equal to twice the width of the edge region for the target segmentation object, determine the edge position information of the target segmentation object on the corresponding pixel row by using the horizontal coordinate at the edge of the target segmentation object on the corresponding pixel row and the length of the edge region;

[0026] If the number of pixels in any of the pixel rows of the target segmentation object is less than twice the width of the edge area for the target segmentation object, determine the vertical coordinate of the middle pixel point of the target segmentation object on the corresponding pixel row; if the number of pixels belonging to the target segmentation object in the pixel column where the vertical coordinate is located is greater than or equal to twice the width of the edge area for the target segmentation object, use the vertical coordinate and the edge area width to determine the edge position information of the target segmentation object on the corresponding pixel column.

[0027] In a possible implementation, converting the semantic segmentation image into pseudo point cloud data for the target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data set includes:

[0028] Mapping each first acquisition point in the three-dimensional point cloud data to the semantic segmentation image, and determining a first mapping point corresponding to each first acquisition point on the semantic segmentation image and pixel position information of the first mapping point;

[0029] Perform nearest neighbor matching on the pixel position information of each pixel point in the semantic segmentation image and the pixel position information of each first mapping point, and select second mapping points that are matched to the corresponding pixel point from each first mapping point;

[0030] Using the depth information of each of the second mapping points matched to the same pixel point, the depth information of the pixel point is obtained;

[0031] Obtaining spatial position information of each pixel in the semantic segmentation image mapped to the three-dimensional point cloud data based on the depth information and pixel position information of each pixel in the semantic segmentation image;

[0032] The semantic information of each pixel in the semantic segmentation image and the mapped spatial position information constitute pseudo point cloud data for the target scene.

[0033] In a possible implementation, the obtaining the semantic information of each of the first acquisition points based on the semantic information mapping contained in the semantic segmentation image includes:

[0034] Mapping each first acquisition point in the three-dimensional point cloud data to the semantic segmentation image, and determining a first mapping point corresponding to each first acquisition point on the semantic segmentation image and pixel position information of the first mapping point;

[0035] Performing nearest neighbor matching on the pixel position information of each of the first mapping points and the pixel position information of each pixel in the semantic segmentation image, and selecting a target pixel corresponding to the first mapping point from each pixel in the semantic segmentation image;

[0036] The semantic information of each of the target pixel points matched to the same first mapping point in the semantic segmentation image is used to obtain the semantic information of the first acquisition point corresponding to the first mapping point.

[0037] In a possible implementation, performing semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image includes:

[0038] Inputting the two-dimensional image into a semantic segmentation model to identify object category information, semantic features and pixel position information of each pixel point contained in at least one segmented object contained in the two-dimensional image;

[0039] Acquire the color value of each pixel point included in the at least one segmentation object in the two-dimensional image;

[0040] Obtaining a semantic segmentation image for the two-dimensional image by using the color value of each pixel point included in the same segmented object, the object category information, the semantic feature, and each pixel position information;

[0041] The color value, the object category information and the semantic feature of each pixel point included in the same segmented object constitute the semantic information of the corresponding pixel point.

[0042] A second aspect of the present application provides a data acquisition device, the device comprising:

[0043] An acquisition module is used to acquire the three-dimensional point cloud data and two-dimensional images collected synchronously from the target scene;

[0044] A semantic segmentation module, used to perform semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image;

[0045] A pseudo point cloud data acquisition module, configured to convert the semantic segmentation image into pseudo point cloud data for the target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data set;

[0046] A data supplementation module, used to supplement the three-dimensional point cloud data corresponding to the target segmentation object by using the pseudo point cloud data for the target segmentation object in the target scene, so as to obtain target point cloud data for the target scene;

[0047] The target point cloud data is used to perform target recognition on the target scene.

[0048] A third aspect of the present application provides an electronic device, the electronic device comprising: at least one communication port, at least one memory and at least one processor, wherein:

[0049] The communication port is used to obtain the three-dimensional point cloud data and the two-dimensional image synchronously collected from the target scene;

[0050] The processor is used to load and execute multiple computer instructions stored in the memory to implement the following steps:

[0051] Performing semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image;

[0052] Converting the semantic segmentation image into pseudo point cloud data for a target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data;

[0053] Using the pseudo point cloud data for the target segmented object in the target scene, the three-dimensional point cloud data corresponding to the target segmented object is supplemented to obtain target point cloud data for the target scene;

[0054] The target point cloud data is used to perform target recognition on the target scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0056] Figure 1 A schematic diagram of a system architecture applicable to the data acquisition method proposed in the embodiment of the present application;

[0057] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in Embodiment 1 of the present application;

[0058] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in Embodiment 2 of the present application;

[0059] Figure 4 A flowchart of a data acquisition method provided in Example 1 of the present application;

[0060] Figure 5 A schematic diagram of a semantic segmentation processing scenario for a two-dimensional image in a data acquisition method provided in an embodiment of the present application;

[0061] Figure 6 A flow chart of a data acquisition method provided in Example 2 of the present application;

[0062] Figure 7 A flowchart of a data acquisition method provided in Example 3 of the present application;

[0063] Figure 8 A flowchart of a data acquisition method provided in Example 4 of the present application;

[0064] Fig. 9 A flowchart of a data acquisition method provided in Example 5 of the present application;

[0065] Fig.10 A schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation mode of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application. The embodiments of the present application are described below in conjunction with the drawings. It is known to those of ordinary skill in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0067] The terms "first", "second" etc. in the full text of the application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable in appropriate circumstances, and this is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0068] The present application can be applied to vehicle network applications such as automatic / assisted driving, environmental perception, rescue services, vehicle management, or other target recognition scenarios based on multi-sensor fusion technology. Before realizing target recognition for the target scene, the data acquisition process for realizing target recognition can execute the data acquisition method proposed in the embodiment of the present application. For the two-dimensional image and three-dimensional point cloud data collected for the same target scene, the depth information contained in the three-dimensional point cloud data is used to convert the semantic segmentation image of the two-dimensional image into pseudo point cloud data, which is equivalent to adding the depth information corresponding to each pixel to the two-dimensional image. Since the semantic segmentation image contains the semantic information of each pixel, the pseudo point cloud data of the target segmentation object can be used to supplement the three-dimensional point cloud data of the same target segmentation object in the initially collected three-dimensional point cloud data to obtain dense target point cloud data. Compared with the global difference supplementation method of three-dimensional point cloud data based on two-dimensional images, this method of improving the density of the three-dimensional point cloud of the target scene saves computing power and reduces the interference features introduced by the supplementation through targeted point cloud supplementation of the target segmentation object, which is conducive to improving the accuracy of target recognition realized for the target scene based on the target point cloud data.

[0069] The following will take the Internet of Vehicles environment perception as an example to introduce the data acquisition method proposed in the embodiment of the present application. It is understandable that the present application can be applied to, but not limited to, electronic devices such as terminal devices or servers (such as cloud servers or physical servers) with data processing capabilities, or through communication between terminal devices and servers to implement the data acquisition method proposed in the present application. The present application does not limit the system architecture of the application scenario.

[0070] Reference Figure 1 , is a schematic diagram of a system architecture applicable to the data acquisition method proposed in the embodiment of the present application, the system may include: a point cloud acquisition device 110, an image acquisition device 120 and an electronic device 130, wherein:

[0071] The point cloud acquisition device 110 can be used to acquire three-dimensional point cloud data (3D Point Cloud Data) of the target scene. The three-dimensional point cloud data can be called point cloud data or point cloud, which is a discrete data set composed of a series of three-dimensional coordinate points. Each coordinate point (i.e., acquisition point) contains the coordinate values ​​of the three coordinate axes of X, Y and Z in the world coordinate system (i.e., ) where the target scene is located (which can be recorded as spatial position information), which is used to describe the surface shape, spatial position, size and other collective features of the objects in the target scene.

[0072] In an embodiment of the present application, the point cloud acquisition device 110 may include one or more of a laser radar, a laser scanner, a depth camera, a structured light scanner, or a robot vision navigation. The embodiment of the present application takes a laser radar as an example of the point cloud acquisition device 110. Multiple laser radars can be deployed for the target scene to collect radar point clouds, i.e., three-dimensional point cloud data, of the target scene. The acquisition implementation process is not described in detail in this application.

[0073] Among them, multiple laser radars can be the same model of equipment from the same manufacturer to avoid the generalization difficulty problem caused by inconsistent configuration parameters of multiple laser radars. In another possible implementation, the present application may not limit the manufacturers to which each laser radar belongs, and can solve the generalization difficulty problem caused by different calibration parameters of multiple laser radars from different manufacturers by eliminating the intensity information (intensity) in the radar point cloud that affects the generalization of multiple laser radars. The present application can replace the intensity information with richer and more clearly textured semantic information obtained from two-dimensional images to improve the accuracy of target recognition based on target point cloud data. The implementation process can refer to the description of the corresponding part of the method embodiment below.

[0074] The image acquisition device 120 can be used to acquire a two-dimensional image of the target scene. In order to reliably implement the data acquisition method proposed in the embodiment of the present application, the image acquisition device 120 and the point cloud acquisition device 110 can be controlled to synchronously acquire data of the target scene to obtain the three-dimensional point cloud data and the two-dimensional image corresponding to the same time. The electronic device 130 can send a data acquisition instruction to the image acquisition device 120 and the point cloud acquisition device 110, so that the image acquisition device 120 and the point cloud acquisition device 110 synchronously acquire the three-dimensional point cloud data and the two-dimensional image of the target scene according to the data acquisition time indicated by the data acquisition instruction, but it is not limited to this data acquisition control method.

[0075] In the embodiments of the present application, the image acquisition device 120 may be a camera, an independent camera, or a terminal device equipped with a camera, such as a drone or a robot. The present application does not limit the product type of the image acquisition device 120. The following embodiments only use the image acquisition device 120 such as a camera as an example for illustration.

[0076] It should be noted that in the multimodal data collection of the target scene in this application, the three-dimensional point cloud data and the two-dimensional image involved in the data fusion need to ensure synchronization in time and space, and the image acquisition device 120 and the point cloud acquisition device 110 can be controlled to collect the corresponding type of data for the target scene at the same frequency, and send it to the electronic device to execute the subsequent steps of the data acquisition method. Based on this, in a possible implementation, the image acquisition device 120 and the point cloud acquisition device 110 can be two independent devices, and the electronic device 130 can realize the data synchronization acquisition control of the image acquisition device 120 and the point cloud acquisition device 110 directly or through other communication devices.

[0077] In another possible implementation, the image acquisition device 120 and the point cloud acquisition device 110 may also be deployed in the same device, such as in the same data acquisition vehicle / drone or other data acquisition device, and in response to a data acquisition instruction input by a user to the data acquisition device, or in response to a data acquisition instruction sent by an external device (such as an electronic device 130 or other remote control device, etc.), the image acquisition device 120 and the point cloud acquisition device 110 are controlled to achieve synchronous acquisition of corresponding types of data of the target scene. The present application does not impose any restrictions on the deployment relationship between the image acquisition device 120 and the point cloud acquisition device 110 and the implementation method of their data acquisition control, which may be determined according to the circumstances.

[0078] The electronic device 130 is respectively connected to the point cloud acquisition device 110 and the image acquisition device 120 for communication, and is used to receive the three-dimensional point cloud data of the target scene acquired by the point cloud acquisition device 110, and the two-dimensional image of the target scene acquired synchronously by the image acquisition device 120, and execute the data acquisition method proposed in the embodiment of the present application. The implementation process can refer to the description of the corresponding part of the method embodiment below, and this embodiment will not be described in detail here.

[0079] In the embodiment of the present application, the electronic device 130 may be a terminal or a server, or may include a terminal and a server, and the server provides the method proposed in the embodiment of the present application to one or more terminals. The present application does not limit the product form of the electronic device 130.

[0080] Among them, the terminal can be installed with an application that can realize target recognition based on point cloud data, or a browser that can access the target recognition platform in the form of a web page. The application and the web page can provide an interactive interface for the user to input relevant parameters on the interactive interface, such as selecting the three-dimensional space range of the target scene where the target recognition task is performed, such as a city street address or a cargo transportation workshop, etc., inputting data acquisition instructions for the image acquisition device 120 and the point cloud acquisition device 110, or inputting data acquisition instructions for the selected target scene, that is, a start instruction for executing the data acquisition method proposed in the embodiment of the present application, etc. The relevant parameters can be sent to the server, and the server obtains the corresponding processing results based on the received parameters, and feeds the processing results back to the terminal.

[0081] For example, the server sends the two-dimensional image and three-dimensional point cloud data of the target scene synchronously collected by the image acquisition device 120 and the point cloud acquisition device 110 to the terminal, and the terminal executes the method provided in this embodiment; or the server executes the method provided in the embodiment of the present application, and feeds back the obtained target point cloud data to the terminal to execute downstream tasks. The server can also further perform target identification based on the target point cloud data based on the task execution instructions of the downstream tasks for the target point cloud data sent by the terminal, such as target identification instructions, and feed back the obtained target identification results (such as the target vehicle's driving route, or traffic flow statistics, etc., which can be determined according to the task content) to the terminal output, etc. The present application does not limit the implementation process of the server cooperating with the terminal to implement the data acquisition method provided in the embodiment of the present application.

[0082] In some other embodiments, the terminal can also obtain the processing result based on the received parameters, such as user input instructions or received collected data, that is, the terminal itself completes the method provided in the embodiment of the present application, and does not require the cooperation of the server to implement, so as to better meet the data acquisition needs in a non-networked environment. The present application does not limit the system architecture composition.

[0083] The terminals in the above embodiments may include but are not limited to mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, robots or industrial computers, etc., which can be determined according to the system configuration of the application scenario. The above servers may be one or more physical servers, or cloud servers with cloud computing capabilities, etc. This application does not limit the product form of the terminal and the type of server, which can be determined according to the situation.

[0084] Combined with the above analysis, refer to Figure 2, is a schematic diagram of the hardware structure of an electronic device provided in the first embodiment of the present application. As described above, the function of the electronic device 130 is described by taking the electronic device 130 as a server. The electronic device 130 may include but is not limited to: at least one communication port 131, at least one memory 132 and at least one processor 133. Among them, at least one communication port 131, at least one memory 132 and at least one processor 133 may communicate through a bus, and the bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one bidirectional line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0085] The communication port 131 can be used to obtain the three-dimensional point cloud data and two-dimensional image synchronously collected for the target scene. In combination with the above description of the data collection process, the communication port 131 may include communication elements corresponding to wireless communication methods such as wifi, global system of mobile communication (Global System of Mobile communication, GSM) communication, general packet radio service (General Packet Radio Service, GPRS), Bluetooth and / or near-field channel methods, so that the electronic device can use the communication element to realize data transmission with other devices (such as the above-mentioned image acquisition device 120 and point cloud acquisition device 110, or other external devices, etc.). This application does not limit the various wireless communication mechanisms supported by the communication port 131. It can be understood that the communication port 131 can also include one or more interfaces that support wired communication methods, such as general-purpose input / output (GPIO) interface, USB interface, universal asynchronous receiver / transmitter (UART) interface, etc., to realize data transmission between various components inside the electronic device.

[0086] The memory 132 can be used to store instructions and data, which can be divided into an instruction storage area and a data storage area. The data storage area can store various data obtained during the execution of the data acquisition method, such as three-dimensional point cloud data, two-dimensional images, target point cloud data, and various intermediate data generated during the execution of the method (such as semantic segmentation images, pseudo point cloud data), etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, and can also include non-volatile random access memory, which provides the processor 133 with hardware, software, and data resources including management of electronic equipment, support for control software and applications, and is also used for storage of running programs and applications.

[0087] The processor 133, as the control center of the electronic device 130, can connect the various components of the entire electronic device 130 through various interfaces and lines, and execute various functions and process data of the electronic device 130 by running or executing computer instructions stored in the memory 132 and calling the stored data. In the embodiment of the present application, the processor 133 loads and executes multiple computer instructions for implementing the data acquisition method provided in the embodiment of the present application to implement the data acquisition method. The implementation process can refer to the description of the corresponding part of the method embodiment below.

[0088] In practical applications, the processor 133 may include one or more processing units, such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), a digital signal processor (DSP), a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a modem processor, or any one or more of the above processors. Optionally, the processor 133 and the memory 132 may be deployed on the same chip or on different independent chips, depending on the circumstances.

[0089] It should be understood that Figure 2 The structure of the electronic device 130 shown does not constitute a limitation on the electronic device 130 in the embodiment of the present application. In practical applications, the electronic device 130 may include Figure 2 More or less components as shown, or some components may be combined. If the electronic device 130 is the above-mentioned terminal, Figure 3As shown, it may also include: one or more components including: a radio frequency component, a camera, an audio circuit, a speaker, a microphone, a headphone jack, an external interface, a power supply, and an input component (such as a touch screen and / or a mouse key, a function key, a joystick or a trackball or other input device), etc. Figure 3 The components listed are not shown. The hardware structure can be determined according to the terminal type and its functional requirements, and this application does not list them one by one here.

[0090] In combination with the above description of the system architecture and the hardware structure of the electronic device, in order to solve the technical problems raised in the background technology part, the data acquisition method proposed in the embodiment of the present application is introduced in detail below with reference to the accompanying drawings.

[0091] Reference Figure 4 , is a flow chart of a data acquisition method provided in Embodiment 1 of the present application, and the method is applicable to an electronic device 130, such as Figure 4 As shown, the data acquisition method may include but is not limited to the following steps:

[0092] Step S401, obtaining three-dimensional point cloud data and two-dimensional images synchronously collected from a target scene;

[0093] According to the relevant description of the system embodiment above, the target scene can be a three-dimensional environment that requires target recognition, such as an urban street scene, a workshop product transportation scene, or other Internet of Vehicles application scenarios. This application does not limit the content of the target scene and its determination method.

[0094] In order to ensure the acquisition consistency between the three-dimensional point cloud data and the two-dimensional image, and to ensure the reliability and accuracy of the multimodal data fusion results, the point cloud acquisition device 110 and the image acquisition device 120 used to realize the synchronous acquisition of the multimodal data of the target scene can be calibrated in advance, such as the laser radar (or laser sensor, etc.) and the camera, to obtain the conversion posture information between the two (that is, the data conversion information between the two coordinate systems), to realize the data conversion between the three-dimensional point cloud data and the two-dimensional image, and meet the subsequent processing requirements. It should be noted that the present application does not elaborate on the method for implementing the position calibration between the point cloud acquisition device 110 and the image acquisition device 120, which can be implemented before determining the types of the two devices and executing the data acquisition method proposed in the present application, thereby completing the synchronous acquisition of the corresponding modal data of the target scene.

[0095] Step S402, performing semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image;

[0096] As described above in relation to the technical solution of the present application, the present application needs to increase the density of the sparse three-dimensional point cloud data directly collected from the target scene, and fuse the more accurate and rich semantic information contained in the two-dimensional image. Compared with the intensity information in the three-dimensional point cloud data, the present application uses the semantic information in the fused dense target point cloud data for target recognition, which not only improves the accuracy of target recognition, but also solves the problem of generalization difficulties caused by different standardization parameters of multiple lidars from different manufacturers, thereby improving the generalization of data acquisition equipment.

[0097] Based on this, the collected two-dimensional image of the target scene can be input into the semantic segmentation network for semantic segmentation to obtain a semantic segmentation image containing the semantic segmentation result. The semantic segmentation network may include but is not limited to ‌ (Fully Convolutional Networks, Fully Convolutional Neural Networks)‌,‌ (Deep Convolutional Network Architecture for Semantic Segmentation), ‌ (Pyramid Scene Parsing Network, which can use the pyramid pooling module to aggregate global information to improve segmentation accuracy), (Multi-path refinement network, combining features at different levels for high-resolution semantic segmentation) or (deep learning model for semantic image segmentation), etc. This application does not restrict the structure of the semantic segmentation network, which can be determined according to the processing requirements of the target scene.

[0098] For example, Figure 5 As shown in the figure, the target scene is a city street. The point cloud and image of the selected city street are collected synchronously to obtain the corresponding 3D point cloud data and 2D image. Figure 5 The two-dimensional image shown is input into the semantic segmentation network, and the category information of each pixel in the two-dimensional image is extracted. Each pixel with the same category information constitutes a segmentation object, and the category information is determined as the object category information of the segmentation object, such as Figure 5 The segmented objects in the image are truck, person, stop sign, sky-other-merged, tree-merged, potted pland, etc. Figure 5The semantic segmentation image shown in the figure can be marked with the segmentation area and object category information corresponding to each segmentation object determined based on the semantic segmentation result. The acquisition process of the semantic segmentation image can be determined in combination with the principle of the corresponding semantic segmentation network. This embodiment will not be described in detail. It should be noted that for each pixel in the two-dimensional image, other attribute information, such as RGB (red, green, and blue) color value, can be identified to enrich the semantic information of the pixel.

[0099] Step S403, converting the semantic segmentation image into pseudo point cloud data for the target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data set;

[0100] Combined with the above description of sparse three-dimensional point cloud data and two-dimensional images, in order to realize the conversion between these two modal (or type) data, the corresponding data acquisition devices are calibrated to obtain the conversion information between the data in their respective coordinate systems, such as based on the laser radar coordinate system (i.e., the point cloud coordinate system for the target scene, which is a global three-dimensional coordinate system, with the laser radar (i.e., the point cloud acquisition device 110) used for point cloud acquisition of the target scene as the origin, the front of it is the X axis, the right side is the Y axis, and the top is the Z axis. (X, Y, Z) can be used to represent the three-dimensional coordinate value of the surface point of the scanned object, i.e., the spatial position information), the camera coordinate system (with the optical center of the camera (image acquisition device 120) as the coordinate origin, the X axis and the Y axis are respectively parallel to the X axis and the Y axis of the image coordinate system, and the optical axis is the Z axis), the image coordinate system (with the center of the image plane as the coordinate origin, the X axis and the Y axis are respectively parallel to the two vertical sides of the image plane, and its coordinate value is represented by (x, y)) and the pixel coordinate system (a two-dimensional coordinate system that can be constructed with a vertex of the image plane as the origin, X The A-axis and Y-axis are respectively parallel to the X-axis and Y-axis of the image coordinate system, and are measured in pixels, with (u, v) used to represent their coordinate values. This application does not elaborate on the method for determining the coordinate transformation relationship.

[0101] In order to specifically improve the density of the directly collected three-dimensional point cloud data of the target scene, reduce or even avoid the introduction of interference features in the process of interpolation and supplementation of the three-dimensional point cloud data, which affects the accuracy of the obtained target point cloud data for target recognition, the present application proposes to utilize the depth information (such as the Z-axis coordinate value in the three-dimensional coordinate value XYZ) possessed by each acquisition point contained in the three-dimensional point cloud data (in order to distinguish the acquisition point of the point cloud data obtained in other processing stages, the acquisition point here can be recorded as the first acquisition point) and the data conversion information between the three-dimensional point cloud data and the semantic segmentation image determined by the above method (such as the above-mentioned coordinate conversion relationship) to obtain the depth information of each pixel in the semantic segmentation image, and determine the semantic segmentation image with added depth information as pseudo point cloud data.

[0102] It can be seen that in the implementation process of step S403, the semantic segmentation image can be integrated with the depth information in the corresponding three-dimensional point cloud data, and the semantic segmentation image can be converted from the two-dimensional pixel coordinate system to the three-dimensional coordinate system (i.e., the point cloud coordinate system or the world coordinate system) to obtain pseudo point cloud data of each pixel point, so that it can be matched with the three-dimensional point cloud data of the target scene directly collected in three dimensions. This application does not elaborate on the data conversion implementation process.

[0103] Step S404, using the pseudo point cloud data of the target segmented object in the target scene, the three-dimensional point cloud data of the corresponding target segmented object is supplemented to obtain the target point cloud data of the target scene; the target point cloud data is used to perform target recognition on the target scene.

[0104] Following the above analysis, the pseudo point cloud data for the target scene obtained by the embodiment of the present application has a higher density than the directly collected three-dimensional point cloud data, and has rich semantic information with clear texture features, which can more accurately distinguish which pseudo point cloud data each segmentation object corresponds to. Figure 5 After obtaining the truck and other vehicle category objects shown in the figure, this part of the pseudo point cloud data can be used to perform local targeted compensation (i.e., the designated area where the target segmented object is located) for the directly collected 3D point cloud data. It is only necessary to supplement the 3D point cloud data of the target segmented object to improve the density of the 3D point cloud data in the area where the target segmented object is located, and there is no need to improve the 3D point cloud data of other segmented objects in the 3D point cloud data of the target scene, i.e., the 3D point cloud data in the area where the non-target segmented objects are located. In addition, since the original collected 3D point cloud data is dense near and sparse farther away, the present application adopts this local (i.e., the designated area where the target segmented object is located) supplementation method of the 3D point cloud data of the target scene to balance accuracy and computing power, reduce the resource consumption of the supplement execution process, reduce the interference of the 3D point cloud data of non-target segmented objects on subsequent target recognition, and improve the efficiency and accuracy of subsequent target recognition.

[0105] In some embodiments, for the pseudo point cloud data for the target scene, the semantic information contained therein is used to extract the pseudo point cloud data in the area where the target segmentation object is located, sample the point cloud data therefrom, and supplement the three-dimensional point cloud data in the area where the target segmentation object is located by interpolation, thereby increasing the amount of point cloud data of the target segmentation object, and also having the semantic information of the corresponding target segmentation object, such as the pixel color value, semantic features, and object category information of the target segmentation object, so that it can more richly and accurately describe the texture features of the target segmentation object. In this way, in the process of using the target point cloud data to identify the target scene, the target segmentation object and its position information can be quickly and accurately determined by identifying the semantic information and depth information of each segmentation object. It can be seen that in the scene of identifying urban road vehicles (i.e., target segmentation objects), through the data acquisition method of the present application, accurate recognition of road vehicles is performed using laser radars and cameras, while enhancing the generalization of multiple laser radar point cloud data.

[0106] To summarize, for the synchronously collected three-dimensional point cloud data and two-dimensional images for the target scene, in the process of performing data fusion on them and improving the point cloud density of the target scene, the embodiment of the present application proposes converting the semantic segmentation image obtained by semantic segmentation of the two-dimensional image into pseudo point cloud data in the three-dimensional coordinate system where the three-dimensional point cloud data is located based on the depth information of each first acquisition point of the three-dimensional point cloud data itself, so that the pseudo point cloud data contains richer semantic information with clear texture features for each pixel point, as well as three-dimensional coordinates for matching three-dimensional point cloud data, and can accurately extract the pseudo point cloud data of the target segmentation object therefrom to accurately supplement the designated area (belonging to the local area) of the sparse three-dimensional point cloud data directly collected for the target scene, that is, to supplement the sparse three-dimensional point cloud data in the area where the target segmentation object is located, to avoid the interference features caused by the point cloud supplementation of the three-dimensional point cloud data of non-target segmentation objects, save computing power, and help improve the accuracy of target recognition (such as environmental perception, etc.) of the target scene.

[0107] Reference Figure 6 , is a flow chart of a data acquisition method provided in Embodiment 2 of the present application. This embodiment can describe a possible detailed implementation of the data acquisition method proposed above, such as Figure 6 As shown, the data acquisition method proposed in this embodiment may include:

[0108] Step S601, obtaining three-dimensional point cloud data and two-dimensional images synchronously collected from a target scene;

[0109] Step S602, performing semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image; the semantic segmentation image contains each segmentation object having semantic information;

[0110] Step S603, converting the semantic segmentation image into pseudo point cloud data for the target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data set;

[0111] Regarding the implementation process of step S601-step S603, you can refer to the description of the corresponding part of the context embodiment, such as the corresponding description content of step S401-step S403, and the embodiment of the present application will not be described in detail here.

[0112] Step S604, determining pseudo point cloud data of at least one target segmentation object according to semantic information contained in the pseudo point cloud data of the target scene;

[0113] In practical applications, if a top view of three-dimensional point cloud data is photographed as a plane, and the point cloud data of the area of ​​interest (such as the area where the target segmented object is located) is extracted from it and feature-stitched and fused with the corresponding area of ​​interest in the two-dimensional image to obtain fused data for target recognition, due to the imaging principle of the image acquisition device itself, there will be interference objects such as background or other non-target objects in each extracted area of ​​interest (such as each segmented object in the target scene). The fused data obtained by direct splicing and fusion contains the point cloud data of the interference object, which will have an adverse effect on subsequent target recognition and reduce the accuracy of target recognition.

[0114] Therefore, an embodiment of the present application proposes to convert a semantically segmented image into pseudo point cloud data for a target scene based on the depth information inherent in the three-dimensional point cloud data. Since each pixel in the pseudo point cloud data (or the acquisition point mapped to the three-dimensional coordinate system where the three-dimensional point cloud data is located, such as a mapped radar point, etc.) has depth information and semantic information, it replaces the two-dimensional image / semantic segmentation image for data fusion with the three-dimensional point cloud data. It is possible to accurately extract the semantic information of the pixel from the pseudo point cloud data of the target scene, indicating the pseudo point cloud data corresponding to the pixel of the target segmentation object, and use it as supplementary point cloud data for improving the density of the three-dimensional point cloud data, so as to avoid the supplementary point cloud data containing pseudo point cloud data of pixel points of non-target segmentation objects (i.e., interference data relative to the corresponding target segmentation object).

[0115] It can be seen that in the process of using dense pseudo point cloud data to encrypt and supplement sparse three-dimensional point cloud data (such as directly collected radar point cloud data), in order to save computing power, all point cloud data in the area of ​​interest (that is, each segmented object) will not be fused. Instead, pseudo point cloud data in the area where the target segmented object is located will be extracted to implement subsequent encryption and supplement steps. There is no need to encrypt and supplement the three-dimensional point cloud data in other non-target segmented areas.

[0116] For example, in Figure 5In the road perception application in the urban street scene shown in the figure, the segmented objects (such as Figure 5 The segmented objects that are not on the road (such as vehicles, pedestrians, etc.) are determined as target segmentation objects; Figure 5 The plants, buildings, etc. shown in the figure are non-target segmentation objects. The present application can extract the pseudo point cloud data corresponding to the vehicles and pedestrians from the pseudo point cloud data of the target scene to encrypt and supplement the point cloud corresponding to the areas where the vehicles and pedestrians are located in the three-dimensional point cloud data of the target scene.

[0117] Step S605, sampling the supplementary point cloud data of the corresponding target segmentation object from the pseudo point cloud data of at least one target segmentation object;

[0118] Step S606, using the supplementary point cloud data of at least one target segmentation object, interpolate and supplement the three-dimensional point cloud data of the corresponding target segmentation object to obtain supplemented three-dimensional point cloud data for the target scene; the spatial position information and semantic information of each first acquisition point in the supplemented three-dimensional point cloud data and each second acquisition point supplemented by the difference constitute the target point cloud data for the target scene, and the semantic information of each first acquisition point is obtained based on the semantic information mapping contained in the semantic segmentation image.

[0119] In practical applications, the more point cloud data that is supplemented to the three-dimensional point cloud data, the greater the probability of introducing interference data, and the more computing resources consumed in the supplementation process, which will also lead to more resource consumption in the target recognition process of the target scene based on the supplemented point cloud data, and even reduce the target recognition efficiency due to excessive calculation. In this regard, the embodiment of the present application proposes to appropriately reduce the amount of point cloud data to be identified while meeting the subsequent target recognition accuracy requirements for the target scene. On the basis of ensuring the accurate three-dimensional point cloud data directly collected, the amount of point cloud data supplemented is appropriately reduced, and the appropriate point cloud density is controlled to reduce the computing resource consumption and calculation amount at each stage in the entire application scenario.

[0120] Therefore, for the pseudo point cloud data corresponding to each determined target segmentation object, some appropriate pseudo point cloud data can be sampled therefrom and determined as the supplementary point cloud data of the corresponding target segmentation object, so as to supplement the sparse three-dimensional point cloud data of the target segmentation object directly collected, so as to reasonably and accurately improve the three-dimensional point cloud data density of the target segmentation object, ensure the subsequent target recognition requirements for the target scene, and reduce the consumption of computing resources. This application does not limit the implementation method of the supplementary point cloud data sampling of step S605.

[0121] For the supplementary point cloud data of the target segmentation object sampled from the pseudo point cloud data of each target segmentation object, it can be supplemented to the three-dimensional point cloud data of the target segmentation object directly collected by interpolation, which can be realized based on the spatial position information of each collection point of the supplementary point cloud data in the three-dimensional coordinate system (i.e., the point cloud coordinate system, such as the laser radar coordinate system, etc.), so as to achieve the purpose of improving the point cloud density with as little computing power as possible. For the convenience of description, the sparse collection points in the three-dimensional point cloud data collected directly are recorded as the first collection points, and the new collection points of the supplementary point cloud data interpolated to the sparse three-dimensional point cloud data are recorded as the second collection points, so that the supplemented three-dimensional point cloud data contains a small number of original first collection points and a large number of second collection points supplemented by interpolation, which is equivalent to the number of collection points of the three-dimensional point cloud data collected directly.

[0122] For each first acquisition point in the directly acquired three-dimensional point cloud data, in order to improve the recognition accuracy using its intensity information, the embodiment of the present application can obtain the semantic information of each first acquisition point based on the semantic information mapped from the semantic segmentation image to replace the intensity information of the first acquisition point, so that each acquisition point in the supplemented three-dimensional point cloud data (the sum of the first acquisition point and the second acquisition point) has corresponding semantic information and spatial position information in the three-dimensional coordinate system (that is, three-dimensional coordinate values, whose Z-axis coordinate values ​​represent depth information). In order to solve the problem of difficulty in generalizing three-dimensional point cloud data between different point cloud acquisition devices, the present application will eliminate the intensity information in the supplemented three-dimensional point cloud data, and only use the semantic information and spatial position information of each first acquisition point and each second acquisition point to constitute the target point cloud data of the target scene.

[0123] In this way, in applications where target recognition is required for target scenes to achieve downstream tasks such as assisted autonomous driving, environmental perception, and vehicle management, the target point cloud data for the target scene obtained according to the data acquisition method proposed in this application is locally densified point cloud data (the number of point clouds in the area where the target segmentation object is located is sufficient, and the number of point clouds in other areas is sparse), and compared with directly collecting global dense three-dimensional point cloud data, each collection point in the target point cloud data contains rich semantic information with clear texture features, and intensity information is no longer used. Target recognition of the target scene is performed based on this, which not only saves computing power, but also greatly improves the accuracy of target recognition.

[0124] Reference Figure 7 , is a flow chart of the data acquisition method provided in the third embodiment of the present application. This embodiment can describe an optional refined implementation method of the data acquisition method proposed above, such as describing a possible implementation method of how to obtain pseudo point cloud data and how to improve the density of three-dimensional point cloud data, such as Figure 7 As shown, the data acquisition method proposed in this embodiment may include:

[0125] Step S701, obtaining three-dimensional point cloud data and two-dimensional images synchronously collected from the target scene;

[0126] Step S702, performing semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image;

[0127] Step S703, mapping each first acquisition point in the three-dimensional point cloud data to a semantic segmentation image, determining a first mapping point corresponding to each first acquisition point on the semantic segmentation image and pixel position information of the first mapping point;

[0128] Since the directly collected three-dimensional point cloud data of the target scene contains a small amount of spatial position information of each of the first acquisition points (i.e., the three-dimensional coordinate value in the three-dimensional coordinate system of the point cloud acquisition device (such as the laser radar coordinate system, which can also be called the point cloud coordinate system), which can also be called the point cloud coordinate, which can be expressed as x, y, z, that is) and intensity information (intensity, which can be recorded as i), the information possessed by each first acquisition point can be expressed as (x, y, z, i). Among them, x can represent the coordinate value on the X-axis of the three-dimensional coordinate system; y can represent the coordinate value on the Y-axis of the three-dimensional coordinate system; z can represent the coordinate value on the Z-axis of the three-dimensional coordinate system, which also represents the depth information (depth) corresponding to the first acquisition point.

[0129] Similarly, the information of each pixel in the semantic segmentation image can be expressed as (u, v, r, g, b, semantic_feature, class), where u and v can represent the pixel position information of the corresponding pixel; r, g, b can represent the color value of the corresponding pixel; semantic_feature can represent the semantic feature of the corresponding pixel identified from the two-dimensional image during the semantic segmentation of the two-dimensional image; class can represent the category information of the corresponding pixel, that is, the object category information of the segmented object to which the pixel belongs. In the embodiment of the present application, the information (r, g, b, semantic_feature, class) can be collectively referred to as the semantic information of the pixel.

[0130] It can be seen that each first acquisition point in the directly collected three-dimensional point cloud data does not have rich semantic information, and the intensity information of the first acquisition point has poor target recognition accuracy relative to the semantic information, and it will cause generalization difficulties between the three-dimensional point cloud data collected by point cloud data devices from different manufacturers. In order to solve this problem, so that each first acquisition point has rich semantic information with clear texture features, the semantic information of each first acquisition point in the synchronously collected three-dimensional point cloud data can be determined based on the semantic information of each pixel point in the semantic segmentation image, and added to the three-dimensional point cloud data, so that the information of each first acquisition point for the target scene can be expressed as (x, y, z, i, r, g, b, semantic_feature, class). This application does not limit the implementation method of how to assign semantic information to the directly collected first acquisition points.

[0131] In some embodiments, taking the synchronous data collection of the target scene by the laser radar and the camera as an example, the position conversion information (such as the conversion matrix) between the respective coordinate systems can be determined based on the position calibration parameters between the laser radar and the camera, such as the external parameter matrix and the internal parameter matrix of the camera calibration, and the three-dimensional point cloud data of the target scene collected by the laser radar is mapped to the two-dimensional coordinate system (such as the image coordinate system / pixel coordinate system) where the semantic segmentation image is located, and the mapping point (referred to as the first mapping point) of each first collection point in the two-dimensional coordinate system is determined, and the spatial position information (such as the three-dimensional coordinate value) of the first collection point in the laser radar coordinate system is converted into the pixel position information (such as the two-dimensional coordinate value) in the two-dimensional coordinate system. The present application does not elaborate on the conversion implementation process.

[0132] Step S704, performing nearest neighbor matching on the pixel position information of each first mapping point and the pixel position information of each pixel in the semantic segmentation image, and selecting a target pixel that matches the corresponding first mapping point from each pixel in the semantic segmentation image;

[0133] Step S705, using the semantic information of each target pixel matched to the same first mapping point in the semantic segmentation image, to obtain the semantic information of the first acquisition point corresponding to the first mapping point;

[0134] Continuing with the above analysis, the nearest neighbor matching method can be used near each first mapping point to select the pixel points (recorded as target pixel points) that are the nearest neighbors to the first mapping point, so as to use the semantic information of these target pixel points to determine the semantic information of the first acquisition point corresponding to the first mapping point, such as assigning the above information (r, g, b, semantic_feature, class) to the first acquisition point corresponding to the first mapping point, so that the existing (x, y, z, i) of the first acquisition point is concatenated with the assigned (r, g, b, semantic_feature, class) to obtain the information represented by (x, y, z, i, r, g, b, semantic_feature, class).

[0135] Among them, in the nearest neighbor matching process, the Euclidean distance measurement method can be used to implement it, such as the two-dimensional Euclidean distance operator and the three-dimensional Euclidean distance operator. Therefore, for each first mapping point, the distance between each pixel and the first mapping point is obtained by the Euclidean distance measurement method. Then, the pixel whose distance is less than the first distance threshold is selected as the target pixel of the nearest neighbor matching of the first mapping point, that is, in the plane / two-dimensional coordinate system where the semantic segmentation image is located, each target pixel within the circular closed range formed by the first distance threshold as the radius with the first mapping point as the center. Then, the average value of the color values ​​(i.e., r, g, b) of these target pixel points can be determined as the color value of the first mapping point, and other contents such as the semantic features and category information in the semantic information of these target pixel points can be determined as the semantic features and category information of the first mapping point. The category information of each target pixel point of the nearest neighbor matching of the same first mapping point is the same. If it is different, the category information with the largest number of corresponding target pixels can be selected as the category information of the first mapping point, and its semantic features can be spliced ​​to determine it as the semantic feature of the first mapping point, but it is not limited to this processing method.

[0136] In another possible implementation, after the nearest neighbor of each first mapping point is matched to each target pixel, the color values ​​of the nearest neighbor matched to each target pixel can be weighted and summed according to the distance between the first mapping point and each target pixel, and then the average value of the total color value is taken as the color value of the first mapping point. The closer the distance between the first mapping point and the target pixel, the closer the semantic information of the target pixel and the first mapping point, the closer the color values ​​of the two, and the greater the weight configured for the color value of the target pixel. This application does not limit the weight value of the color value of each target pixel.

[0137] It should be noted that the implementation method of how to use the semantic information of each pixel in the semantic segmentation image to configure semantic information for each first acquisition point in the three-dimensional point cloud data includes but is not limited to the nearest neighbor matching implementation method described above. In addition, in some target recognition tasks, for the three-dimensional point cloud data directly collected, the existing intensity information of each first acquisition point can still be used as semantic information, combined with the pseudo point cloud data fusion method described below, to obtain the target point cloud data for implementing the target recognition task, and is not limited to the method described by steps S703-S705, which assigns semantic information to each first acquisition point in the three-dimensional point cloud data based on the semantic information in the semantic segmentation image. After executing step S703, step S706 can be executed directly. The execution process of the data acquisition method described in this embodiment is not described in detail in this application.

[0138] Step S706, performing nearest neighbor matching on the pixel position information of each pixel point in the semantic segmentation image and the pixel position information of each first mapping point, and selecting, from each first mapping point, each second mapping point to which the corresponding pixel point matches;

[0139] Step S707, using the depth information of each second mapping point matched to the same pixel point, to obtain the depth information of the pixel point;

[0140] As analyzed above, the two-dimensional image / semantic segmentation image of the target scene obtained by the present application has pixel position information and semantic information for each pixel, but does not have depth information, such as the coordinate value on the Z axis in the laser radar coordinate system. The embodiment of the present application will use the depth information of each first acquisition point of the three-dimensional point cloud data itself to obtain the depth information of each pixel of the semantic segmentation image after the semantic segmentation of the two-dimensional image. In order to ensure the accuracy of the depth information of each pixel as much as possible, after mapping the three-dimensional point cloud data to the plane / two-dimensional coordinate system where the semantic segmentation image is located according to the method described above, for each pixel in the semantic segmentation image, the depth information of the second mapping point matched to the nearest neighbor of the pixel can be selected to obtain the depth information of the pixel, such as performing an average operation on the depth information of each second mapping point matched to the nearest neighbor of the pixel, and determining the obtained average depth value as the depth value of the pixel.

[0141] In another possible implementation, for each pixel point, the nearest neighbor matches each second mapping point, and the depth information of each second mapping point matched by the nearest neighbor can be weighted and summed according to the distance between the second mapping point and the pixel point (such as the Euclidean distance calculated by the Euclidean distance measurement method, or other distance measurement methods), and then the average value of the total depth information is taken as the depth information of the pixel point. Among them, the closer the distance between the second mapping point and the pixel point, the greater the weight configured for the depth information of the second mapping point, and the greater the influence on the depth information of the pixel point. This application does not limit the weight value of the depth information of each second mapping point.

[0142] Step S708, obtaining spatial position information of each pixel in the semantic segmentation image mapped to the three-dimensional point cloud data based on the depth information and pixel position information of each pixel in the semantic segmentation image; the semantic information of each pixel in the semantic segmentation image and the mapped spatial position information constitute pseudo point cloud data for the target scene;

[0143] Since the depth information refers to the coordinate value on the Z axis of the three-dimensional coordinate system, the pixel position information includes the coordinate values ​​of the pixel point on the X axis and Y axis of the two-dimensional coordinate system (such as the image / pixel coordinate system) of the semantic segmentation image, and the pixel position information of the pixel point is represented by pixels, which can be represented as (u, v), where u can represent the pixel coordinate on the X axis of the two-dimensional coordinate system, and v can represent the pixel coordinate on the Y axis of the two-dimensional coordinate system. The pixel position information and depth information of the pixel point can be converted based on the position conversion information (such as the coordinate conversion matrix) between the two-dimensional coordinate system and the three-dimensional coordinate system, and (u, v) and the depth information (i.e., the depth value depth) can be converted into the three-dimensional coordinate value in the three-dimensional coordinate system, i.e., the spatial position information, which can be represented as (x, y, z), where x can represent the coordinate value on the X axis of the three-dimensional coordinate system, which is converted from u; y can represent the coordinate value on the Y axis of the three-dimensional coordinate system, which is converted from v; z can represent the coordinate value on the Z axis of the three-dimensional coordinate system, which is converted from depth. This application does not describe the conversion implementation process in detail.

[0144] After the conversion processing described above, the spatial position information (x, y, z) of each pixel in the semantic segmentation image is mapped to the three-dimensional coordinate system, and the existing semantic information (r, g, b, semantic_feature, class) of each pixel constitutes the pseudo point cloud data of the corresponding pixel in the three-dimensional coordinate system, which can be represented by (x, y, z, r, g, b, semantic_feature, class). Compared with the three-dimensional point cloud data of each first acquisition point directly collected, that is, represented by (x, y, z, i), the semantic information of the pseudo point cloud data is richer in semantic content and clearer in texture features than the corresponding mapped acquisition point represented by the intensity information i (intensity).

[0145] For example, the urban road perception scenario is still used as an example for explanation. Figure 8 As shown, the three-dimensional point cloud data (i.e. Figure 8 The original three-dimensional point cloud data in the image is sparse point cloud data, which contains a small number of first acquisition points. If it is directly used for target recognition of the target scene, it is easy to cause low accuracy due to too few feature content, affecting the reliability and accuracy of the target recognition task. Therefore, the embodiment of the present application proposes to synchronously collect a two-dimensional image (i.e., a large number of pixel points and rich semantic information) of the target scene. Figure 8 The original two-dimensional image in the image is converted into pseudo point cloud data containing the depth information of each pixel and the existing semantic information according to the method described above, and then fused with the original three-dimensional point cloud data, that is, the three-dimensional point cloud data is used as the reference, and the pseudo point cloud data is used to supplement the sparse positions of the three-dimensional point cloud data to improve the density of the sparse three-dimensional point cloud data. The density improvement process is implemented as follows.

[0146] It should be understood that in the data acquisition process, if the method described above is followed, the three-dimensional point cloud data after assigning semantic information to each first acquisition point based on the semantic information of each pixel in the semantic segmentation image is represented as: (x, y, z, i, r, g, b, semantic_feature, class), but the amount of pseudo point cloud data is much smaller, and pseudo point cloud data needs to be used to encrypt and supplement it. It should be noted that this application does not restrict the execution order between the process of obtaining pseudo point cloud data and the implementation process of assigning semantic information to each first acquisition point originally collected in the three-dimensional point cloud data. It can be executed synchronously or sequentially, and is not limited to the execution order described in the steps of this embodiment.

[0147] In some other embodiments, for the method of obtaining pseudo point cloud data, the present application can also directly map the semantic segmentation image to the three-dimensional coordinate system where the three-dimensional point cloud data is located. For each third mapping point after the mapping of each pixel point, the first acquisition point that is the nearest neighbor match of the third mapping point can be selected from each first acquisition point, and the depth information of the third mapping point can be obtained by using the depth information of the third mapping point, so that each third mapping point has spatial position information and semantic information to constitute pseudo point cloud data. The implementation process is similar to the mapping conversion process described in the above steps, and this application will not describe it in detail.

[0148] Step S709, determining pseudo point cloud data of at least one target segmentation object according to semantic information contained in the pseudo point cloud data for the target scene;

[0149] In an embodiment of the present application, in order to save computing power and reduce the interference point cloud data introduced in the process of increasing the density of three-dimensional point cloud data, which is equivalent to integrating all pseudo point cloud data of the target scene into sparse three-dimensional point cloud data, the present application proposes to encrypt and supplement the sparse three-dimensional point cloud data of the designated area in the target scene, i.e., the area where the target segmentation object is located. To this end, the pseudo point cloud data corresponding to each target segmentation object in the pseudo point cloud data of the target scene can be identified based on the object category information (such as the above-mentioned class) in the semantic information of each pixel point contained in the pseudo point cloud data.

[0150] Step S710, determining a non-edge region of at least one target segmentation object; the non-edge region refers to a region where the accuracy of depth information mapped by each pixel point of the corresponding target segmentation object is greater than or equal to a threshold;

[0151] Step S711, sampling from the pseudo point cloud data located in the non-edge area of ​​at least one target segmentation object to obtain the supplementary point cloud data of the corresponding target segmentation object;

[0152] Step S712, using the supplementary point cloud data of at least one target segmentation object, interpolate and supplement the three-dimensional point cloud data of the corresponding target segmentation object to obtain target point cloud data for the target scene; the target point cloud data is composed of spatial position information and semantic information corresponding to each first acquisition point and each interpolated and supplemented second acquisition point.

[0153] In some embodiments, for the pseudo point cloud data of each determined target segmentation object, a number of pseudo point cloud data can be randomly sampled as supplementary point cloud data for the target segmentation object, and supplemented into the three-dimensional point cloud data of the target scene corresponding to the area where the target segmentation object is located. The area where the point cloud data to be supplemented is located in the original three-dimensional point cloud data, that is, the three-dimensional point cloud data of the area where the target segmentation object is located, can be determined based on the position mapping relationship between the two.

[0154] Preferably, if the method described above is used to assign semantic information to each first acquisition point in the original three-dimensional point cloud data, the three-dimensional point cloud data of each target segmentation object can be determined based on the semantic information of each first acquisition point, and after determining the supplementary point cloud data of each target segmentation object, the three-dimensional point cloud data and the supplementary point cloud data of the same target segmentation object can be fused based on the semantic information of each first acquisition point in the determined three-dimensional point cloud data and the semantic information of each pixel point in the supplementary point cloud data to obtain the target point cloud data of the target segmentation object. However, it is not limited to the point cloud data fusion implementation method described in this application.

[0155] From the above description of the process of obtaining pseudo point cloud data, it can be seen that in the process of obtaining the depth information of the pixel points on the edge of the target segmentation object, since the first mapping points of its nearest neighbor matching not only include the first mapping points belonging to the target segmentation object, but also include the first mapping points that do not belong to the target segmentation object, the depth information of the pixel points determined by averaging the depth information of these first mapping points is not accurate enough, which means that there are large errors in the content of each pseudo point cloud data on the edge of each target segmentation object in the pseudo point cloud data. If random sampling is used to obtain the supplementary point cloud data of the target segmentation object, and the three-dimensional point cloud data of the target segmentation object is encrypted and supplemented, the error caused by the inaccurate edge pseudo point cloud data to the target point cloud data cannot be avoided, which further affects the accuracy of target recognition of the target scene.

[0156] In order to improve the above problems and further reduce interference with point cloud data, the present application configures an edge danger area for the target segmentation object to achieve random sampling of pseudo point cloud data in the target segmentation object area (i.e., non-edge area) outside the edge danger area, thereby ensuring the accuracy of the obtained supplementary point cloud data. Therefore, the present application can analyze the accuracy of the depth information mapped to each pixel point contained in the target segmentation object in the semantic segmentation image to extract the edge danger area with an accuracy less than a threshold, and use the remaining area in the target segmentation object as a non-edge area, which is equivalent to redetermining the edge of the target segmentation object, and obtaining pseudo point cloud data containing pixels with a depth information accuracy greater than or equal to the threshold. The threshold can be determined based on experience or experiment, and the present application does not limit the size of the threshold.

[0157] For example, Figure 8 The data acquisition method flow chart shown in FIG. 1 shows a method for converting a semantic segmentation image into pseudo point cloud data for a target scene according to the above steps S703, S706-S708. However, due to inaccurate depth information of pixels on the edge of the segmented object, the following problems may occur: Figure 5The edge of the truck in the pseudo point cloud data shown is unclear / fuzzy. If the supplementary point cloud data of the truck is randomly sampled, the point cloud data at the blurred position of the truck may be sampled and supplemented to the three-dimensional point cloud data. This will cause the spatial position information of the second acquisition point in the obtained target point cloud data of the truck to be inaccurate (which is caused by inaccurate depth information of the second acquisition point). In this way, in the process of target recognition of the target scene based on the target point cloud data of the target scene, the target recognition accuracy will be reduced due to inaccurate spatial position information.

[0158] In this regard, the embodiment of the present application can redefine the edge of the target segmentation object according to the method described above, determine a non-edge area (which is equivalent to the non-edge area of ​​the edge obtained after the original semantic segmentation) that does not contain pixels with inaccurate depth information, and usually reduce the area of ​​the target segmentation object to obtain a non-edge area of ​​the target segmentation object with clear edges. The present application does not limit the implementation method of step S710. In this way, in the case of Figure 8 In the flowchart shown, after obtaining the pseudo point cloud data, the above-mentioned step S710 can be added, so that in the point cloud fusion process, as described in step S710, the pseudo point cloud data in the re-determined non-edge area of ​​the target segmentation object can be randomly sampled to obtain the supplementary point cloud data of the target segmentation object, and supplemented to the non-edge area of ​​the target segmentation object in the original three-dimensional point cloud data, thereby increasing the point cloud density in the non-edge area, avoiding the introduction of inaccurate point cloud data of the target segmentation object, and improving the accuracy of the target point cloud data.

[0159] Among them, the present application assigns semantic information to each first acquisition point in the original three-dimensional point cloud data according to the above method, and adopts the above point cloud density improvement method to supplement the point cloud data of the target segmentation object, eliminates the existing intensity information of each first acquisition point, and uses the spatial position information and semantic information of each first acquisition point and each second acquisition point supplemented by interpolation to form the target point cloud data, which can be expressed as: (x, y, z, r, g, b, semantic_feature, class), that is, the semantic information from the two-dimensional image is used to replace the intensity information i of the point cloud acquisition device (such as the semantic information of the lidar), and the target point cloud data is input into the target recognition network, so as to accurately obtain the target recognition result of the target scene, such as Figure 8 The urban road perception results under the target scenario shown are used to assist in directing road traffic or assist in guiding the vehicle's autonomous driving route, etc.

[0160] Moreover, since the intensity information i of the point cloud acquisition device is calibrated differently by different point cloud acquisition device manufacturers, its numerical distribution is also different. It is difficult to generalize the data acquisition method by using point cloud acquisition devices from multiple different manufacturers to collect three-dimensional point cloud data for the target scene. In order to avoid the occurrence of generalization difficulties, the target point cloud data obtained by this application can delete the intensity information in the supplemented three-dimensional point cloud data, and only use the semantic information from the two-dimensional image for road perception. Combined with the more sophisticated point cloud data fusion strategy described above, it reduces computing power and greatly improves road perception accuracy.

[0161] Reference Fig. 9 , which is a flow chart of a data acquisition method provided in Embodiment 5 of the present application. This embodiment can describe a possible implementation method of how to determine the non-edge area of ​​each target segmentation object in the data acquisition method described in the above embodiment. For the implementation process of other steps of the data acquisition method, reference can be made to the description of the corresponding part of the context embodiment, which will not be described in detail in this embodiment. Fig. 9 As shown, the process of determining the non-edge area of ​​the target segmentation object may include:

[0162] Step S901, determining the number of pixels of the same target segmentation object contained in each pixel row in the pseudo point cloud data of the target scene;

[0163] Step S902, determining edge position information of the corresponding target segmentation object in the corresponding pixel row according to the determined number of pixels of each target segmentation object and the edge area width of the corresponding target segmentation object;

[0164] Step S903 , using the edge position information of each target segmentation object in different pixel rows, determine the non-edge area of ​​the corresponding target segmentation object.

[0165] According to the method described in the above embodiment, after the semantic segmentation image of the target scene is converted into pseudo point cloud data, in order to ensure the accuracy of the supplementary point cloud data of each target segmentation object sampled, the non-edge area of ​​each target segmentation object in the pseudo point cloud data for the target scene can be re-determined, and the edge area with inaccurate depth information can be eliminated.

[0166] Based on this, in the edge area design scheme proposed in the embodiment of the present application, each row of pixels can be scanned to calculate the number of pixels occupied by the target segmentation object in the pixel row, that is, the number of pixels occupied by the mask (mask, which is the classification result at the pixel level) after semantic segmentation, that is, the number of pixels in this pixel row that are located in the same mask (that is, belong to the same target segmentation object) (recorded as the number of pixels). Afterwards, based on the number of pixels and the pre-configured edge area width of the target segmentation object (which can be determined based on experience or experiments, or configured in combination with the object category and its vertical coordinate, or dynamically adjusted according to actual conditions, and this application does not impose any restrictions on its numerical value), the edge position information of the target segmentation object in the pixel row can be reasonably adjusted, usually moving it to the center position of the target segmentation object, and this application does not impose any restrictions on the implementation process.

[0167] In one possible implementation, since each row of pixels scanned will pass through the edge of the target segmentation object twice (i.e., the left edge and the right edge), each edge needs to adjust an edge area width. Therefore, in the implementation process of step S902, the number of pixels of the same target segmentation object in any scanned pixel row can be compared with twice the edge area width for the target segmentation object. If the number of pixels of any pixel row of the target segmentation object is greater than or equal to twice the edge area width for the target segmentation object, the horizontal coordinate at the edge of the target segmentation object on the corresponding pixel row and the length of the edge area can be used to determine the edge position information of the target segmentation object on the corresponding pixel row.

[0168] After the above comparison, if the number of pixels in any pixel row of the target segmentation object is less than twice the width of the edge area for the target segmentation object, determine the vertical coordinate of the middle pixel point of the target segmentation object on the corresponding pixel row; if the number of pixels belonging to the target segmentation object in the pixel column where the vertical coordinate is located is greater than or equal to twice the width of the edge area for the target segmentation object, use the vertical coordinate and the edge area width to determine the edge position information of the target segmentation object on the corresponding pixel column.

[0169] For example, if the number of pixels occupied by the mask in a row of pixels currently scanned is greater than twice its edge area, the edge area of ​​the corresponding target segmentation object is formed with the horizontal coordinate (X-axis coordinate) at the edge as the center and the edge area length as the window width, and the new edge position information of the target segmentation object in the row of pixels is obtained. If the number of pixels occupied by the mask is less than twice the width of the edge area, the vertical coordinate (Y-axis coordinate) of the middle position of the mask in the pixel row can be calculated to determine the number of pixels occupied by the mask edge on the Y-axis (i.e., the pixel column where the vertical coordinate is located). If the number of pixels is greater than or equal to twice the width of the edge area, the mask edge area is re-determined on the Y-axis, that is, the edge position information of the corresponding target segmentation object on the pixel column is re-determined. If the number of pixels is less than twice the edge width, the edge of the corresponding target segmentation object does not need to be processed, that is, the processing of the mask is abandoned.

[0170] It should be understood that in another possible implementation, for the process of determining the non-edge area of ​​the target segmentation object, the number of pixels of the same target segmentation object contained in each pixel column in the pseudo point cloud data for the target scene can also be determined by scanning each column of pixels, and the edge position information of the corresponding target segmentation object in the corresponding pixel column is determined based on the determined number of pixels of each target segmentation object and the edge area width for the corresponding target segmentation object, thereby using the edge position information of each target segmentation object in different pixel columns to determine the non-edge area of ​​the corresponding target segmentation object. This implementation process is similar to the determination process described in the above scanning each row of pixels, and this application will not be described in detail.

[0171] Afterwards, the present application re-determines the non-edge area of ​​the target segmentation object according to the above method for the pseudo point cloud data obtained for the target scene, indicating that the pseudo point cloud data in the non-edge area is accurate enough, and the supplementary point cloud data of the target segmentation object can be randomly sampled or otherwise sampled therefrom, and fused into the area where the target segmentation object is located in the original three-dimensional point cloud data, thereby realizing point cloud encryption and supplementation in the specified area, reliably improving the accuracy of the target point cloud data, thereby improving the accuracy of target recognition of the target scene, and greatly reducing computing power.

[0172] For the data acquisition method described in the above embodiments, in the process of semantically segmenting the two-dimensional image collected from the target scene to obtain the corresponding semantic segmentation image, an existing semantic segmentation model (such as the semantic segmentation network listed above) can be called to implement it, and the synchronously collected two-dimensional image can be input into the semantic segmentation model to identify the object category information (class), semantic features (semantic_feature) and pixel position information of each pixel point contained in at least one segmentation object contained in the two-dimensional image (such as the above-mentioned two-dimensional pixel coordinates, u, v). In addition, the color value (such as rgb) of each pixel point contained in at least one segmentation object in the two-dimensional image can be obtained, so as to obtain the semantic segmentation image for the two-dimensional image by using the color value, object category information, semantic features and pixel position information of each pixel point contained in the same segmentation object; wherein the color value, object category information and semantic features of each pixel point contained in the same segmentation object constitute the semantic information of the corresponding pixel point, which can be expressed as: (r, g, b, semantic_feature, class).

[0173] It should be noted that the semantic information content of each pixel in the semantic segmentation image includes but is not limited to the multi-dimensional information listed above. In some tasks, YUV values ​​can also be added or YUV values ​​can be used to replace RGB values, etc., which can be determined according to actual conditions. In the process of obtaining electronic device display data, the object segmentation results and object category information can be annotated to the image, such as Figure 5 and Figure 7 The semantic segmentation image shown is not limited to the content marked in the figure.

[0174] A data acquisition method provided by an embodiment of the present application is introduced above, and a device for executing the above data acquisition method will be introduced below.

[0175] Reference Fig.10 , is a schematic diagram of the structure of a data acquisition device provided in an embodiment of the present application. Fig.10 As shown, the data acquisition device may include:

[0176] An acquisition module 1010 is used to acquire the three-dimensional point cloud data and the two-dimensional image synchronously collected from the target scene;

[0177] A semantic segmentation module 1020, configured to perform semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image;

[0178] A pseudo point cloud data acquisition module 1030 is used to convert the semantic segmentation image into pseudo point cloud data for the target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data set;

[0179] A data supplementation module 1040 is used to supplement the three-dimensional point cloud data corresponding to the target segmented object by using the pseudo point cloud data for the target segmented object in the target scene, so as to obtain target point cloud data for the target scene;

[0180] The target point cloud data is used to perform target recognition on the target scene.

[0181] In some embodiments, since the semantic segmentation image contains various segmentation objects with semantic information, the data supplementation module 1040 may include:

[0182] A first determining unit, configured to determine the pseudo point cloud data of at least one target segmentation object according to semantic information contained in the pseudo point cloud data for the target scene;

[0183] A sampling unit, configured to sample the supplementary point cloud data corresponding to the target segmentation object from the pseudo point cloud data of each of the at least one target segmentation object;

[0184] an interpolation and supplementation unit, configured to interpolate and supplement the three-dimensional point cloud data corresponding to the target segmented object using the supplementary point cloud data of the at least one target segmented object, so as to obtain supplemented three-dimensional point cloud data for the target scene;

[0185] Among them, the spatial position information and semantic information of each of the first acquisition points in the supplemented three-dimensional point cloud data and each of the second acquisition points supplemented by the difference constitute the target point cloud data for the target scene; the semantic information of each of the first acquisition points is obtained based on the semantic information mapping contained in the semantic segmentation image.

[0186] In a possible implementation, the sampling unit may include:

[0187] A non-edge region determining unit, configured to determine a non-edge region of each of the at least one target segmentation object; the non-edge region refers to a region where the accuracy of depth information mapped by each pixel point of the corresponding target segmentation object is greater than or equal to a threshold;

[0188] The supplementary point cloud data sampling unit is used to sample the supplementary point cloud data corresponding to the target segmentation object from the pseudo point cloud data located in the non-edge area of ​​each of the at least one target segmentation objects.

[0189] Optionally, the non-edge area determining unit may include:

[0190] A second determining unit is used to determine the number of pixels of the same target segmentation object contained in each pixel row or pixel column in the pseudo point cloud data;

[0191] A third determining unit, configured to determine edge position information of the corresponding target segmentation object in the corresponding pixel row or pixel column according to the determined number of pixels of each target segmentation object and the edge area width of the corresponding target segmentation object;

[0192] The fourth determining unit is used to determine the non-edge area of ​​the corresponding target segmentation object by using the edge position information of each target segmentation object in different pixel rows or pixel columns.

[0193] Optionally, the third determining unit may include at least one of the following determining subunits:

[0194] A first determining subunit is configured to determine edge position information of the target segmentation object on the corresponding pixel row using the horizontal coordinate at the edge of the target segmentation object on the corresponding pixel row and the length of the edge region if the number of pixels of any pixel row of the target segmentation object is greater than or equal to twice the width of the edge region for the target segmentation object;

[0195] The second determination subunit is used to determine the vertical coordinate of the middle pixel point of the target segmentation object on the corresponding pixel row if the number of pixels in any of the pixel rows of the target segmentation object is less than twice the width of the edge area for the target segmentation object, and if the number of pixels belonging to the target segmentation object in the pixel column where the vertical coordinate is located is greater than or equal to twice the width of the edge area for the target segmentation object, determine the edge position information of the target segmentation object on the corresponding pixel column using the vertical coordinate and the edge area width.

[0196] In some other embodiments proposed in the present application, the pseudo point cloud data acquisition module 1030 may include:

[0197] a pixel position information determining unit, configured to map each first acquisition point in the three-dimensional point cloud data to the semantic segmentation image, and determine a first mapping point corresponding to each first acquisition point on the semantic segmentation image and pixel position information of the first mapping point;

[0198] A second mapping point selection unit is used to perform nearest neighbor matching on the pixel position information of each pixel point in the semantic segmentation image and the pixel position information of each first mapping point, and select second mapping points corresponding to the pixel point from each first mapping point;

[0199] A depth information obtaining unit, configured to obtain depth information of the pixel point by using the depth information of each of the second mapping points matched to the same pixel point;

[0200] A spatial position information obtaining unit, configured to obtain spatial position information of each pixel in the semantic segmentation image mapped to the three-dimensional point cloud data according to the depth information and pixel position information of each pixel in the semantic segmentation image;

[0201] The semantic information of each pixel in the semantic segmentation image and the mapped spatial position information constitute pseudo point cloud data for the target scene.

[0202] In some further embodiments, the semantic information obtaining module for obtaining the semantic information of each of the first acquisition points based on the semantic information mapping contained in the semantic segmentation image may include:

[0203] a fifth determining unit, configured to map each first acquisition point in the three-dimensional point cloud data to the semantic segmentation image, and determine first mapping points corresponding to each first acquisition point on the semantic segmentation image and pixel position information of the first mapping points;

[0204] a target pixel point selection unit, configured to perform nearest neighbor matching on the pixel position information of each of the first mapping points and the pixel position information of each pixel point in the semantic segmentation image, and select a target pixel point that matches the first mapping point from each pixel point in the semantic segmentation image;

[0205] The semantic information obtaining unit is used to obtain the semantic information of the first acquisition point corresponding to the first mapping point by using the semantic information of each target pixel point matched to the same first mapping point in the semantic segmentation image.

[0206] In some further embodiments, the semantic segmentation module 1020 may include:

[0207] A recognition unit, used to input the two-dimensional image into a semantic segmentation model, and recognize object category information, semantic features, and pixel position information of each pixel point contained in at least one segmented object contained in the two-dimensional image;

[0208] A color value acquisition unit, used to acquire the color value of each pixel point included in the at least one segmentation object in the two-dimensional image;

[0209] A semantic segmentation image obtaining unit, used to obtain a semantic segmentation image for the two-dimensional image by using the color value of each pixel point contained in the same segmented object, the object category information, the semantic feature and each pixel position information;

[0210] The color value, the object category information and the semantic feature of each pixel point included in the same segmented object constitute the semantic information of the corresponding pixel point.

[0211] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any data acquisition method provided in the embodiment of the present application.

[0212] When loading and executing computer-readable instructions on a computer, all or part of the process or function according to the embodiment of the present application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer-readable instructions can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium. For example, the computer-readable instructions can be transmitted from a website site, a computer, a training device or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, training device or data center.

[0213] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any data acquisition method provided in the embodiment of the present application.

[0214] The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a training device or a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0215] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0216] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0217] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. In this specification, each embodiment is described in a progressive or parallel manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. For the device and electronic device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description, and this application does not give examples one by one.

Claims

1. A data acquisition method, the method comprising: Acquire the three-dimensional point cloud data and two-dimensional images collected synchronously for the target scene; Performing semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image; Converting the semantic segmentation image into pseudo point cloud data for the target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data set; Using the pseudo point cloud data for the target segmented object in the target scene, the three-dimensional point cloud data corresponding to the target segmented object is supplemented to obtain target point cloud data for the target scene; The target point cloud data is used to perform target recognition on the target scene.

2. According to the method of claim 1, the semantic segmentation image contains each segmentation object with semantic information; the using the pseudo point cloud data for the target segmentation object in the target scene to supplement the three-dimensional point cloud data corresponding to the corresponding target segmentation object to obtain the target point cloud data for the target scene comprises: Determining the pseudo point cloud data of at least one target segmentation object according to the semantic information contained in the pseudo point cloud data for the target scene; Sampling the supplementary point cloud data corresponding to the target segmentation object from the pseudo point cloud data of each of the at least one target segmentation object; Using the supplementary point cloud data of the at least one target segmentation object, interpolate and supplement the three-dimensional point cloud data corresponding to the target segmentation object to obtain supplemented three-dimensional point cloud data for the target scene; The spatial position information and semantic information of each of the first acquisition points in the supplemented three-dimensional point cloud data and each of the second acquisition points supplemented by the difference constitute the target point cloud data for the target scene; The semantic information of each of the first acquisition points is obtained based on the semantic information mapping contained in the semantic segmentation image.

3. The method according to claim 2, wherein the step of sampling the supplementary point cloud data of the corresponding target segmentation object from the pseudo point cloud data of the at least one target segmentation object comprises: Determining a non-edge region of each of the at least one target segmentation objects; The non-edge area refers to an area where the accuracy of the depth information mapped by each pixel point of the corresponding target segmentation object is greater than or equal to a threshold; The supplementary point cloud data corresponding to the target segmentation object is sampled from the pseudo point cloud data located in the respective non-edge area of ​​the at least one target segmentation object.

4. The method according to claim 3, wherein determining the non-edge area of ​​each of the at least one target segmentation objects comprises: Determine the number of pixels of the same target segmentation object contained in each pixel row or pixel column in the pseudo point cloud data; Determine edge position information of the corresponding target segmentation object in the corresponding pixel row or pixel column according to the determined number of pixels of each target segmentation object and the edge area width of the corresponding target segmentation object; The non-edge area of ​​the corresponding target segmentation object is determined by using the edge position information of each target segmentation object in different pixel rows or pixel columns.

5. The method according to claim 4, wherein the step of determining the edge position information of the corresponding target segmentation object in the corresponding pixel row or pixel column based on the determined number of pixels of each target segmentation object and the edge area width of the corresponding target segmentation object comprises at least one of the following: If the number of pixels of any of the pixel rows of the target segmentation object is greater than or equal to twice the width of the edge region for the target segmentation object, determine the edge position information of the target segmentation object on the corresponding pixel row by using the horizontal coordinate at the edge of the target segmentation object on the corresponding pixel row and the length of the edge region; If the number of pixels in any of the pixel rows of the target segmentation object is less than twice the width of the edge area for the target segmentation object, determine the vertical coordinate of the middle pixel point of the target segmentation object on the corresponding pixel row; if the number of pixels belonging to the target segmentation object in the pixel column where the vertical coordinate is located is greater than or equal to twice the width of the edge area for the target segmentation object, use the vertical coordinate and the edge area width to determine the edge position information of the target segmentation object on the corresponding pixel column.

6. The method according to any one of claims 1 to 5, wherein the step of converting the semantic segmentation image into pseudo point cloud data for the target scene based on the depth information of each first acquisition point in the three-dimensional point cloud data set comprises: Mapping each first acquisition point in the three-dimensional point cloud data to the semantic segmentation image, and determining a first mapping point corresponding to each first acquisition point on the semantic segmentation image and pixel position information of the first mapping point; Perform nearest neighbor matching on the pixel position information of each pixel point in the semantic segmentation image and the pixel position information of each first mapping point, and select second mapping points that are matched to the corresponding pixel point from each first mapping point; Using the depth information of each of the second mapping points matched to the same pixel point, the depth information of the pixel point is obtained; Obtaining spatial position information of each pixel in the semantic segmentation image mapped to the three-dimensional point cloud data based on the depth information and pixel position information of each pixel in the semantic segmentation image; The semantic information of each pixel in the semantic segmentation image and the mapped spatial position information constitute pseudo point cloud data for the target scene.

7. The method according to any one of claims 2 to 5, wherein obtaining the semantic information of each of the first acquisition points based on the semantic information mapping contained in the semantic segmentation image comprises: Mapping each first acquisition point in the three-dimensional point cloud data to the semantic segmentation image, and determining a first mapping point corresponding to each first acquisition point on the semantic segmentation image and pixel position information of the first mapping point; Performing nearest neighbor matching on the pixel position information of each of the first mapping points and the pixel position information of each pixel in the semantic segmentation image, and selecting a target pixel corresponding to the first mapping point from each pixel in the semantic segmentation image; The semantic information of each of the target pixel points matched to the same first mapping point in the semantic segmentation image is used to obtain the semantic information of the first acquisition point corresponding to the first mapping point.

8. The method according to any one of claims 1 to 5, wherein the performing semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image comprises: Inputting the two-dimensional image into a semantic segmentation model to identify object category information, semantic features and pixel position information of each pixel point contained in at least one segmented object contained in the two-dimensional image; Acquire the color value of each pixel point included in the at least one segmentation object in the two-dimensional image; Obtaining a semantic segmentation image for the two-dimensional image by using the color value of each pixel point included in the same segmented object, the object category information, the semantic feature, and each pixel position information; The color value, the object category information and the semantic feature of each pixel point included in the same segmented object constitute the semantic information of the corresponding pixel point.

9. A data acquisition device, comprising: An acquisition module is used to acquire the three-dimensional point cloud data and two-dimensional images collected synchronously from the target scene; A semantic segmentation module, used to perform semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image; A pseudo point cloud data acquisition module, configured to convert the semantic segmentation image into pseudo point cloud data for the target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data set; A data supplementation module, used to supplement the three-dimensional point cloud data corresponding to the target segmentation object by using the pseudo point cloud data for the target segmentation object in the target scene, so as to obtain target point cloud data for the target scene; The target point cloud data is used to perform target recognition on the target scene.

10. An electronic device, comprising: at least one communication port, at least one memory, and at least one processor, wherein: The communication port is used to obtain the three-dimensional point cloud data and the two-dimensional image synchronously collected from the target scene; The processor is used to load and execute multiple computer instructions stored in the memory to implement the following steps: Performing semantic segmentation on the two-dimensional image to obtain a semantic segmentation image of the two-dimensional image; Converting the semantic segmentation image into pseudo point cloud data for a target scene according to the depth information of each first acquisition point in the three-dimensional point cloud data; Using the pseudo point cloud data for the target segmented object in the target scene, the three-dimensional point cloud data corresponding to the target segmented object is supplemented to obtain target point cloud data for the target scene; The target point cloud data is used to perform target recognition on the target scene.

Images