Scanning data generation method, electronic device and storage medium

Abstract

The present application provides a method for generating scan data, an electronic device, and a storage medium, the method comprising: acquiring an image and an initial point cloud of a scene; performing object segmentation on the image to obtain a mask map, the mask map including an object region corresponding to an object in the image; projecting the initial point cloud onto the mask map to obtain a first point cloud corresponding to the object region; denoising the first point cloud to obtain a second point cloud; deleting the second point cloud from the initial point cloud to obtain scan data corresponding to the scene. The above method can improve the accuracy of removing point clouds corresponding to dynamic objects.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a scanning data generating method, an electronic device and a storage medium. Background Art

[0002] In the fields of autonomous driving, map navigation and mobile robots, building three-dimensional maps is an important technical link. Smart terminals (such as mobile robots) perceive surrounding objects through sensors and create a three-dimensional map of the environment in which the smart terminal is currently located.

[0003] When a smart terminal is scanning a large scene, moving dynamic objects will affect the accuracy of the scanned object, resulting in more noisy point clouds being scanned. In order to solve this problem, related technologies usually determine the dynamic object by comparing the differences between adjacent frames, thereby eliminating the scanning points corresponding to the dynamic object. This type of method requires adjusting parameters according to changes in the scene and is not universal. In addition, related technologies also use a global point cloud judgment method. When the global point cloud determines that there is a dynamic object, the point cloud in the area where the dynamic object is located is directly deleted. This type of method is prone to accidental deletion. Summary of the invention

[0004] The embodiments of the present application disclose a scanning data generation method, an electronic device and a storage medium, which solve the technical problem of the related art that the accuracy of removing point clouds corresponding to dynamic objects is low.

[0005] The present application provides a method for generating scan data, the method comprising: acquiring an image and an initial point cloud of a scene; performing object segmentation on the image to obtain a mask map, the mask map comprising an object area corresponding to an object in the image; projecting the initial point cloud onto the mask map to acquire a first point cloud corresponding to the object area; denoising the first point cloud to obtain a second point cloud; deleting the second point cloud from the initial point cloud to obtain scan data corresponding to the scene.

[0006] In some embodiments of the present application, the electronic device includes a scanning device, and the projecting of the initial point cloud onto the mask image to obtain a first point cloud corresponding to the object area includes: determining a first image coordinate of the object area in an image coordinate system, and determining a first point cloud coordinate of the initial point cloud in a first scanning module coordinate system; converting the first point cloud coordinates into second point cloud coordinates in a second scanning module coordinate system according to an external parameter matrix of the scanning device that scans the scene; converting the first image coordinates into second image coordinates in a second scanning module coordinate system according to an internal parameter matrix of the scanning device; determining contour coordinates of outer contour points of the object area from the second image coordinates; and determining the first point cloud based on the contour coordinates and the second point cloud coordinates.

[0007] In some embodiments of the present application, determining the first point cloud based on the contour coordinates and the second point cloud coordinates includes: determining the contour position of the outer contour point in the second scanning module coordinate system based on the contour coordinates; connecting the outer contour points based on the contour position to generate an area boundary of the object area in the second scanning module coordinate system; determining the point cloud position of the initial point cloud in the second scanning module coordinate system based on the second point cloud coordinates; and determining the first point cloud based on the point cloud position and the area boundary.

[0008] In some embodiments of the present application, determining the first point cloud based on the point cloud position and the area boundary includes: determining a boundary point cloud located on the area boundary and an in-boundary point cloud located on the inner side of the area boundary based on the point cloud position; and using the boundary point cloud and the in-boundary point cloud as the first point cloud.

[0009] In some embodiments of the present application, projecting the initial point cloud onto the mask image to obtain the first point cloud corresponding to the object area includes: matching the second point cloud coordinates with the second image coordinates to obtain a matching result; and using the initial point cloud corresponding to the successfully matched second point cloud coordinates as the first point cloud.

[0010] In some embodiments of the present application, denoising the first point cloud to obtain the second point cloud includes: determining the noise point cloud in the first point cloud based on a preset neighborhood search algorithm; and removing the noise point cloud from the first point cloud to obtain the second point cloud.

[0011] In some embodiments of the present application, the noise point cloud in the first point cloud is determined based on a preset neighborhood search algorithm, including: taking each point cloud in the first point cloud as a center point; determining the adjacent point cloud of each center point based on the center point and a preset distance threshold, and the adjacent point cloud belongs to the first point cloud; if the number of adjacent point clouds of any center point is less than a preset number threshold, determining any center point as the noise point cloud.

[0012] In some embodiments of the present application, the electronic device includes a first scanning module and a second scanning module, and acquiring an image and an initial point cloud of a scene includes: acquiring an image of the scene through the second scanning module; acquiring the initial point cloud of the scene through the first scanning module; wherein, the first scanning module acquires the image and the second scanning module acquires the initial point cloud simultaneously.

[0013] The present application also provides an electronic device, which includes a processor and a memory, and the processor is used to implement the scanning data generating method when executing a computer program stored in the memory.

[0014] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the scanning data generating method is implemented.

[0015] In the scanning data generation method of the present application, an image and an initial point cloud of a scene are acquired, and the image is segmented to obtain a mask map, which provides a basis for the subsequent deletion of the initial point cloud corresponding to the object in the image. In order to determine the point cloud corresponding to the object area in the initial point cloud, the initial point cloud is projected to the mask map to determine the first point cloud projected to the object area. Since the first point cloud is the result of combining the initial point cloud and the mask map, in order to avoid accidental deletion, the first point cloud is denoised. The denoised second point cloud is obtained, and the second point cloud can accurately describe the location of the object. After the second point cloud is determined, the second point cloud is deleted from the initial point cloud to obtain the scanning data corresponding to the scene, thereby achieving the purpose of deleting the point cloud corresponding to the object from the initial scanning data and avoiding the point cloud of the object in the scene from affecting the scanning accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an application scenario diagram of the scanning data generation method provided in an embodiment of the present application.

[0017] Figure 2 It is a flowchart of the scanning data generation method provided in an embodiment of the present application.

[0018] Figure 3 It is a schematic diagram of generating scanning data provided in an embodiment of the present application.

[0019] Figure 4 This is a flowchart for obtaining the first point cloud provided in an embodiment of the present application.

[0020] Figure 5 This is a mask diagram provided in an embodiment of the present application.

[0021] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given by way of example for reference.

[0023] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0024] In the fields of autonomous driving, map navigation and mobile robots, building three-dimensional maps is an important technical link. Smart terminals (such as mobile robots) perceive surrounding objects through sensors and create a three-dimensional map of the environment in which the smart terminal is currently located.

[0025] When a smart terminal is scanning a large scene, moving dynamic objects will affect the accuracy of the scanned object, resulting in more noisy point clouds being scanned. In order to solve this problem, related technologies usually determine the dynamic object by comparing the differences between adjacent frames, thereby eliminating the scanning points corresponding to the dynamic object. This type of method requires adjusting parameters according to changes in the scene and is not universal. In addition, related technologies also use a global point cloud judgment method. When the global point cloud determines that there is a dynamic object, the point cloud in the area where the dynamic object is located is directly deleted. This type of method is prone to accidental deletion.

[0026] In order to solve the technical problem that the accuracy of the point cloud corresponding to the dynamic object is not high in the related technology, the present application proposes a scanning data generation method, an electronic device and a storage medium, which uses an image to screen the initial point cloud, thereby avoiding the situation of mistaken deletion of the point cloud, and improving the accuracy of the point cloud corresponding to the dynamic object. In order to better understand the scanning data generation method, electronic device and storage medium described in the present application, the application scenario of the present application is first described.

[0027] Figure 1 : is an application scenario diagram of the scanning data generation method provided in the embodiment of the present application. Figure 1 As shown, the scanning data generation method described in the present application is applied to an electronic device 10, and the electronic device 10 includes at least one scanning device 100. The first scanning module 110 and the second scanning module 120 can be integrated in the same scanning device 100, respectively serving as two components of the scanning device 100. In addition, the first scanning module 110 and the second scanning module 120 can also be respectively operated in different scanning devices 100. The present application does not limit the device form, operation mode, etc. of the first scanning module 110 and the second scanning module 120.

[0028] The electronic device 10 may be a computer device, a mobile phone, a tablet computer, a personal digital assistant (PDA), a cloud server, a physical server, etc. The embodiment of the present application does not impose any limitation on the specific type of the electronic device.

[0029] The scanning device 100 may be a tracking scanner, and the first scanning module 110 and the second scanning module 120 may be two sub-devices of the tracking scanner, which may be assembled to operate in coordination, or disassembled to operate independently.

[0030] The first scanning module 110 may be a laser radar scanner, a tracking scanning device, or the like.

[0031] The second scanning module 120 may be a camera or a computer device with an image acquisition function. The present application does not limit the types of the first scanning module 110 and the second scanning module 120 .

[0032] In some embodiments of the present application, the electronic device 10 processes the image captured by the second scanning module 120 and the initial point cloud scanned by the first scanning module 110, so that the initial point cloud can be filtered using the image to obtain scanning data of the scene.

[0033] The indication Figure 1 These are merely examples of application scenarios and do not constitute a limitation on the application scenarios. They may include more or fewer components than those shown in the figures, or a combination of certain components, or different components. For example, the electronic device 10 may also include input and output devices, network access devices, and a power supply.

[0034] Figure 2 is a flowchart of a method for generating scan data provided by an embodiment of the present application, which is applied to an electronic device (eg Figure 1 According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0035] Step S201, obtaining an image and an initial point cloud of a scene.

[0036] In some embodiments of the present application, the scene may be a variety of complex and diverse shooting application scenes, such as different scenes such as indoor, outdoor, and environment. The scene is photographed by the second scanning module to obtain an image corresponding to the scene. The initial point cloud can be obtained by scanning the scene by the first scanning module.

[0037] In order to improve the accuracy of subsequent processing, the second scanning module and the first scanning module can be fixed at the same position during operation. It can be understood that the first scanning module and the second scanning module at the same position have a certain relative distance. The shooting of the scene by the second scanning module and the scanning of the scene by the first scanning module can be performed synchronously. In an example, assuming that the second scanning module triggers the shooting operation of the shooting scene at the first moment, the first scanning module will also trigger the scanning operation of the scanning scene at the first moment, and the subsequent frequency will also remain the same.

[0038] In some embodiments of the present application, the second scanning module and the first scanning module may be connected in communication, and the second scanning module and the first scanning module may determine whether to acquire an image of the scene and an initial point cloud through an interactive instruction. The interactive instruction may be an instruction triggered by a user on the second scanning module or the first scanning module, which is used to instruct the second scanning module to capture the scene and the first scanning module to scan the scene.

[0039] In one example, the second scanning module detects that the user has triggered the shooting button and generates an interaction instruction. The second scanning module sends the interaction instruction to the first scanning module, so that the first scanning module starts scanning in response to the interaction instruction. In another example, the first scanning module detects that the user has triggered the scanning button and generates an interaction instruction. The first scanning module sends the interaction instruction to the second scanning module, so that the second scanning module starts shooting in response to the interaction instruction.

[0040] In some embodiments of the present application, the electronic device can control the second scanning module and the first scanning module to run simultaneously. For example, the electronic device simultaneously sends a shooting instruction to the second scanning module and a scanning instruction to the first scanning module, so that the second scanning module shoots the scene in response to the shooting instruction to obtain an image, and the first scanning module scans the scene in response to the scanning instruction to obtain an initial point cloud.

[0041] In some other embodiments of the present application, the second scanning module may have a video recording function, and the first scanning module may also have a real-time scanning function. In order to facilitate the subsequent processing of the image and the initial point cloud collected at the same time, after the second scanning module captures the image and the first scanning module scans the initial point cloud, the electronic device receives the image and the initial point cloud and can carry the timestamp of the collection. Through the timestamp, the image and the initial point cloud collected at the same time are obtained to improve the accuracy of subsequent processing.

[0042] Step S202, performing object segmentation on the image to obtain a mask map.

[0043] In some embodiments of the present application, object segmentation may also be referred to as image segmentation. Image segmentation is the technology and process of dividing an image into several specific regions with unique properties and proposing objects of interest. Existing image segmentation methods are mainly divided into the following categories: threshold-based segmentation methods, region-based segmentation methods, edge-based segmentation methods, and segmentation methods based on specific theories.

[0044] In order to achieve image segmentation, an algorithm model can be used to process the image. The algorithm model can include any one or more combinations of models such as Long Short-Term Memory (LSTM), Recurrent Neural Network (RNN), and Convolutional Neural Networks (CNN).

[0045] In the field of image processing, masking refers to using a selected image, graphic or object to block the processed image (all or part) to control the image processing area or processing process. Masks have the following four main functions. (1) Extracting the region of interest: multiplying the pre-made region of interest mask with the image to be processed to obtain the region of interest image. The image value in the region of interest remains unchanged, while the image value outside the region is 0. (2) Shielding effect: use a mask to shield certain areas on the image so that they do not participate in the processing or the calculation of the processing parameters, or only process or count the shielded area. (3) Structural feature extraction: use similarity variables or image matching methods to detect and extract structural features in the image that are similar to the mask. (4) Production of special shape images. In the embodiments of the present application, the purpose of using a mask is to extract the region of interest where the object is located.

[0046] After the electronic device acquires the image, it can call the pre-stored algorithm model to segment the image, so as to obtain the mask map corresponding to the object in the image. Among them, the object in the image can be a person or an animal, and this application is not limited to the play. The mask map can be presented in a binary form, and the area of ​​interest in the mask map can be represented by 1, and the area of ​​non-interest can be represented by 0. The area of ​​interest can be the object area where the object is located.

[0047] Step S203: project the initial point cloud onto the mask image to obtain a first point cloud corresponding to the object area.

[0048] In some embodiments of the present application, an image coordinate system corresponding to the mask image is established. A second scanning module coordinate system of the second scanning module and a first scanning module coordinate system of the first scanning module are obtained. According to the coordinate conversion relationship between the image coordinate system, the second scanning module coordinate system and the first scanning module coordinate system, the initial point cloud is projected onto the mask image. The mask image includes an object area, so that the point cloud corresponding to the object area is deleted later. The point cloud projected to the object area is obtained as the first point cloud.

[0049] Step S204: denoising the first point cloud to obtain a second point cloud.

[0050] In some embodiments of the present application, in order to avoid the situation of accidental deletion, a preset neighborhood search algorithm can be used to denoise the first point cloud. Among them, the neighborhood search algorithm can include three categories. The first type of neighborhood search algorithm can be an algorithm based on statistical characteristics, such as statistical outlier removal (SOR) and a neighborhood search algorithm based on KD-Tree. The second type of neighborhood search algorithm can be an algorithm based on space division, such as voxel grid filter and bin denoising algorithm. The third type of neighborhood search algorithm can be an algorithm based on geometric characteristics, such as bilateral filtering algorithm, radius filtering algorithm, etc. The present application does not limit the type of neighborhood search algorithm.

[0051] The noise point cloud in the first point cloud is searched based on the neighborhood search algorithm. Specifically, the first point cloud may be a general term for multiple point clouds. Each point cloud in the first point cloud is taken as a center point, and the neighboring point cloud of each center point is determined based on the distance threshold between the center point and a preset center point. The distance between the center point and other point clouds is calculated. If the distance is less than or equal to the distance threshold, and the other point clouds belong to the first point cloud, the other point clouds are taken as the neighboring point clouds of the center point.

[0052] After the adjacent point clouds of each point cloud in the first point cloud are determined, if the number of adjacent point clouds of any center point is less than a preset number threshold, the center point is determined to be a noise point cloud.

[0053] In one example, it is calculated that point cloud A has 3 adjacent point clouds, and point cloud B has 7 adjacent point clouds. The preset number threshold is 6, then point cloud A is a noise point cloud, and point cloud B is a non-noise point cloud.

[0054] In some embodiments of the present application, after the noise point cloud is determined, the noise point cloud can be removed from the first point cloud. In addition, if the first point cloud belongs to the noise data in the initial point cloud, and the noise point cloud belongs to the noise data of the first point cloud, then the noise point cloud determined from the first point cloud does not belong to the noise data of the first point cloud. Therefore, removing the noise point cloud from the first point cloud is actually deleting the noise mark of the noise point cloud, not removing the noise point cloud.

[0055] Step S205 , deleting the second point cloud from the initial point cloud to obtain scan data corresponding to the scene.

[0056] In some embodiments of the present application, since dynamic objects such as pedestrians may exist during the scanning process of the scene, such dynamic objects may affect the accuracy of the point cloud corresponding to the scanned scene. Therefore, in order to avoid the noise data corresponding to such dynamic point clouds affecting the tracking accuracy of the scene, such noise data may be deleted.

[0057] In some embodiments of the present application, the second point cloud is noise data finally determined from the initial point cloud, which is an interference point cloud generated when an object (e.g., a pedestrian) is identified. Therefore, after the second point cloud is determined, the second point cloud is deleted from the initial point cloud, so that the scanning data corresponding to the scene can be obtained.

[0058] Through the above-mentioned embodiments, an image and an initial point cloud of a scene are acquired, and the image is segmented to obtain a mask map, which provides a basis for the subsequent deletion of the initial point cloud corresponding to the object in the image. In order to determine the point cloud corresponding to the object area in the initial point cloud, the initial point cloud is projected to the mask map to determine the first point cloud projected to the object area. Since the first point cloud is the result of combining the initial point cloud and the mask map, in order to avoid accidental deletion, the first point cloud is denoised. The denoised second point cloud is obtained, and the second point cloud can accurately describe the location of the object. After the second point cloud is determined, the second point cloud is deleted from the initial point cloud to obtain the scanning data corresponding to the scene, thereby achieving the purpose of deleting the point cloud corresponding to the object from the initial scanning data, and avoiding the point cloud of the object in the scene from affecting the scanning accuracy.

[0059] In addition, this type of scan data can also be presented to the user through the display screen of the electronic device. Through the above embodiment, the visual effect of the scan data can be improved.

[0060] Figure 3 Schematic diagram of generating scanning data provided by the embodiment of the present application. Figure 3 As shown, the image captured by the second scanning module 120 and the initial point cloud scanned by the first scanning module 110 are obtained (by Figure 3 The electronic device uses the pre-stored algorithm model to segment the image and obtains the following: Figure 3 The mask image shown in FIG. 1 includes the object area. The initial point cloud is projected onto the mask image to obtain a projection image. As shown in the projection image, the point cloud projected onto the object area is marked to obtain a first point cloud in the projection image. The first point cloud is denoised to obtain a second point cloud. The second point cloud is deleted from the initial point cloud to obtain scan data. Figure 3 The scan data shown corresponds to a schematic presentation.

[0061] Figure 4 is a flowchart of obtaining the first point cloud provided by the embodiment of the present application. After the initial point cloud is projected onto the mask image, the first point cloud corresponding to the object area can be obtained. Figure 4 As shown, the following steps are included:

[0062] Step S401 , determining first image coordinates of an object area in an image coordinate system, and determining first point cloud coordinates of an initial point cloud in a first scanning module coordinate system.

[0063] In some embodiments of the present application, the first image coordinates may include the coordinates of the entire object area in the image coordinate system, for example, the coordinates of a point on the boundary of the object area, or the coordinates of any point in the object area. The first scanning module coordinate system may be a coordinate system provided by the first scanning module, and the first point cloud coordinates of the initial point cloud in the first scanning module coordinate system are obtained.

[0064] Step S402 : converting the first point cloud coordinates into second point cloud coordinates in a second scanning module coordinate system according to an external parameter matrix of a scanning device for scanning a scene.

[0065] In some embodiments of the present application, the extrinsic parameter matrix is ​​usually composed of a 3×3 rotation matrix and a 3×1 translation vector, and the extrinsic parameter matrix is ​​used to describe the position and posture relationship between the second scanning module and the first scanning module, that is, the position and posture relationship between the second scanning module coordinate system and the first scanning module coordinate system. The extrinsic parameter matrix of the scanning device can be the extrinsic parameter matrix of the first scanning module or the extrinsic parameter matrix of the second scanning module.

[0066] The electronic device knows the relative position between the second scanning module and the first scanning module, and can determine the external parameter matrix. Using the external parameter matrix, the first point cloud coordinates are transformed to the second scanning module coordinate system by rotation and translation to obtain the second point cloud coordinates of the initial point cloud in the second scanning module coordinate system.

[0067] Step S403: converting the first image coordinates into second image coordinates in a second scanning module coordinate system according to an intrinsic parameter matrix of the scanning device.

[0068] In some embodiments of the present application, the scanning device includes a second scanning module. The intrinsic parameter matrix includes the focal length, principal point coordinates, and pixel size of the second scanning module. The intrinsic parameter matrix is ​​used to project points in a three-dimensional space onto a two-dimensional image plane. Therefore, according to the intrinsic parameter matrix of the second scanning module, the first image coordinates can be converted into second image coordinates in the coordinate system of the second scanning module, so that the mask image and the initial point cloud are located in the same coordinate system.

[0069] Step S404: determining the contour coordinates of the outer contour points of the object area from the second image coordinates.

[0070] In some embodiments of the present application, the object area is composed of a plurality of outer contour points, and in order to subsequently determine whether there is an overlap relationship with the second point cloud coordinates, the contour coordinates are determined from the second image coordinates.

[0071] Step S405 , determining the first point cloud based on the contour coordinates and the second point cloud coordinates.

[0072] In some embodiments of the present application, the contour position of the outer contour point on the second scanning module coordinate system is determined based on the contour coordinates. The outer contour points are connected based on the contour position to generate a region boundary of the object region on the second scanning module coordinate system. The point cloud position of the initial point cloud on the second scanning module coordinate system is determined based on the second point cloud coordinates. According to the point cloud position, a boundary point cloud located on the region boundary and an in-bounds point cloud located on the inner side of the region boundary can be determined. The boundary point cloud and the in-bounds point cloud are used as the first point cloud.

[0073] like Figure 5 As shown, the initial point cloud is projected onto the mask image, and there are three point clouds D1, D2, and D3. Among them, D1 is located on the region boundary, so D1 can be divided into the first point cloud. D2 is located outside the region boundary, so D2 is not divided into the first point cloud. D3 is located inside the region boundary, so D3 can be divided into the first point cloud.

[0074] In other embodiments of the present application, after the second point cloud coordinates and the second image coordinates are determined, the second point cloud coordinates and the second image coordinates can be matched, so that the first point cloud can be determined from the initial point cloud based on the matching result. The initial point cloud corresponding to the successfully matched second point cloud coordinates is used as the first point cloud.

[0075] Through the above embodiment, the first point cloud corresponding to the object area can be accurately obtained, providing an accurate data basis for subsequent determination of the scan data, and avoiding the situation of accidental deletion to a certain extent, thereby improving the accuracy of removing the point cloud corresponding to the dynamic object.

[0076] Figure 6The scanning data generation method provided in the embodiment of the present application is applied to an electronic device 10, which may be a computer device with a shooting function such as a mobile phone, a tablet computer, or a laptop computer.

[0077] The electronic device 10 includes a communication module 101, a memory 102, a processor 103, an input / output (I / O) interface 104, and a bus 105. The processor 103 is coupled to the communication interface 101, the memory 102, and the I / O interface 104 through the bus 105.

[0078] The communication module 101 may include a wired communication module and / or a wireless communication module. The wired communication module may provide one or more wired communication solutions such as Universal Serial Bus (USB), Controller Area Network (CAN), etc. The wireless communication module may provide one or more wireless communication solutions such as Wireless Fidelity (Wi-Fi), Bluetooth (BT), mobile communication network, Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), etc.

[0079] The memory 102 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The random access memory can be directly read and written by the processor 103, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data. The random access memory may include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), etc.

[0080] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 103. The non-volatile memory may include a disk storage device and a flash memory.

[0081] The memory 102 is used to store one or more computer programs. The one or more computer programs are configured to be executed by the processor 103. The one or more computer programs include multiple instructions. When the multiple instructions are executed by the processor 103, the method for generating scan data of multiple devices executed on the electronic device 10 can be implemented.

[0082] In other embodiments, the electronic device 10 further includes an external memory interface for connecting to an external memory to expand the storage capacity of the electronic device 10 .

[0083] The processor 103 may include one or more processing units, for example, the processor 103 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0084] The processor 103 provides computing and control capabilities. For example, the processor 103 is used to execute a computer program stored in the memory 102 to implement the above-mentioned method for generating scan data for multiple devices.

[0085] The I / O interface 104 is used to provide a channel for user input or output. For example, the I / O interface 104 can be used to connect various input and output devices, such as a mouse, keyboard, touch device, display screen, etc., so that the user can enter information or visualize information.

[0086] The bus 105 is at least used to provide a channel for mutual communication among the communication module 101 , the memory 102 , the processor 103 , and the I / O interface 104 in the electronic device 10 .

[0087] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the methods in the above-mentioned embodiments of the present application.

[0088] The computer-readable storage medium may be an internal memory of the electronic device described in the above embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0089] In some embodiments, the computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0090] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0091] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0092] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0093] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A scanning data generation method, applied to an electronic device, characterized in that: The electronic device includes a scanning device, and the method includes: Get the image and initial point cloud of the scene; Performing object segmentation on the image to obtain a mask image, wherein the mask image includes an object area corresponding to the object in the image; Projecting the initial point cloud onto the mask image to obtain a first point cloud corresponding to the object area includes: determining first image coordinates of the object area in an image coordinate system, and determining first point cloud coordinates of the initial point cloud in a first scanning module coordinate system; converting the first point cloud coordinates into second point cloud coordinates in a second scanning module coordinate system according to an external parameter matrix of the scanning device that scans the scene; converting the first image coordinates into second image coordinates in a second scanning module coordinate system according to an internal parameter matrix of the scanning device; determining contour coordinates of outer contour points of the imaged area from the second image coordinates; and determining the first point cloud based on the contour coordinates and the second point cloud coordinates; De-noising the first point cloud to obtain a second point cloud; The second point cloud is deleted from the initial point cloud to obtain scan data corresponding to the scene.

2. The scanning data generation method according to claim 1, characterized in that: The determining the first point cloud based on the contour coordinates and the second point cloud coordinates includes: Based on the contour coordinates, determining the contour position of the outer contour point on the second scanning module coordinate system; Connecting the outer contour points based on the contour positions to generate a region boundary of the object region in the second scanning module coordinate system; Determine the point cloud position of the initial point cloud in the second scanning module coordinate system based on the second point cloud coordinates; The first point cloud is determined based on the point cloud position and the region boundary.

3. The scanning data generating method according to claim 2, characterized in that: The determining the first point cloud based on the point cloud position and the region boundary includes: Based on the point cloud positions, determining a boundary point cloud located on the boundary of the region and an in-bounds point cloud located inside the boundary of the region; The boundary point cloud and the in-boundary point cloud are used as the first point cloud.

4. The scanning data generation method according to claim 1, characterized in that: The projecting the initial point cloud onto the mask image to obtain a first point cloud corresponding to the object area includes: Matching the second point cloud coordinates with the second image coordinates to obtain a matching result; The initial point cloud corresponding to the second point cloud coordinates that are matched successfully is used as the first point cloud.

5. The scanning data generation method according to claim 1, characterized in that: The step of denoising the first point cloud to obtain a second point cloud includes: Determine a noise point cloud in the first point cloud based on a preset neighborhood search algorithm; The noise point cloud is removed from the first point cloud to obtain the second point cloud.

6. The scanning data generating method according to claim 5, characterized in that: The determining the noise point cloud in the first point cloud based on a preset neighborhood search algorithm includes: Taking each point cloud in the first point cloud as a center point; Determine a neighboring point cloud of each center point based on a distance threshold between the center point and a preset distance threshold, wherein the neighboring point cloud belongs to the first point cloud; If the number of adjacent point clouds of any central point is less than a preset number threshold, the central point is determined to be the noise point cloud.

7. The scanning data generating method according to claim 1, characterized in that: The electronic device includes a first scanning module and a second scanning module, and the acquiring of an image and an initial point cloud of a scene includes: Acquire an image of the scene by the second scanning module; Acquire the initial point cloud of the scene by the first scanning module; Wherein, the first scanning module acquires the image and the second scanning module acquires the initial point cloud synchronously.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the scanning data generating method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the scanning data generating method according to any one of claims 1 to 7 is implemented.

Images