Point cloud and image fusion method and device

Abstract

The present application discloses a method and device for fusing point clouds and images, which are helpful to improve the accuracy of the acquired fused point clouds. The method for fusing point clouds and images is applied to a computer device, and the method includes: obtaining a first point cloud. The first point cloud is a point cloud used to describe a first scene in an autonomous driving environment. The first point cloud includes a first sub-point cloud and a second sub-point cloud, and the first sub-point cloud and the second sub-point cloud are used to describe a first sub-scene and a second sub-scene in the first scene, respectively. Obtain a first image; wherein the first image is an image used to describe the first scene. Divide the first image into a plurality of sub-images. The plurality of sub-images include a first sub-image and a second sub-image, and the first sub-image and the second sub-image are used to describe a first sub-scene and a second sub-scene, respectively. Divide each of the plurality of sub-images into a plurality of image blocks, and fuse the first point cloud and the first image based on the plurality of image blocks.

Description

Technical Field

[0001] The present application relates to the technical field of data fusion, and in particular to a method and device for fusing point cloud and image. Background Art

[0002] In order to meet the needs of environmental perception for autonomous driving, the combination of lidar (lidar light detection and ranging) and cameras for environmental detection has become one of the main forms of environmental detection. Figure 1 As shown, an environment detection system including a camera, a laser radar and a computer device, for the same scene (such as any scene in a natural environment, the scene includes trees, electric poles, street lights, traffic signs and lane lines, etc.), the camera obtains an image, and the laser radar obtains a point cloud, which is composed of multiple scanning points. The computer device receives the image and the point cloud, and divides the image into multiple image blocks of the same size. The computer device fuses the corresponding scanning points with the image blocks to obtain a fused point cloud. The fused point cloud is used for navigation and obstacle avoidance of the autonomous driving car.

[0003] Currently, computer equipment divides an image into multiple image blocks of the same size based on the entire image, which can easily cause matching errors and / or ghosting, or even matching errors in the fused point cloud. This results in poor accuracy of the fused point cloud obtained by the computer equipment, which is not conducive to the navigation and obstacle avoidance of autonomous vehicles. Summary of the invention

[0004] The embodiments of the present application provide a point cloud and image fusion method and device, which can reduce the matching error and / or ghosting generated by the fused point cloud, thereby improving the accuracy of the acquired fused point cloud.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for fusing a point cloud and an image is provided, which is applied to a computer device, and the method includes: obtaining a first point cloud. The first point cloud is a point cloud used to describe a first scene in an autonomous driving environment. The first point cloud includes a first sub-point cloud and a second sub-point cloud, the first sub-point cloud and the second sub-point cloud are used to describe a first sub-scene and a second sub-scene in the first scene, respectively, and the first sub-scene and the second sub-scene are sub-scenes under a first field of view partition and a second field of view partition of a point cloud acquisition device, respectively. Obtain a first image. The first image is an image used to describe the first scene. Divide the first image into a plurality of sub-images. The plurality of sub-images include a first sub-image and a second sub-image, the first sub-image and the second sub-image are used to describe the first sub-scene and the second sub-scene, respectively. Divide each of the plurality of sub-images into a plurality of image blocks. Based on the plurality of image blocks, fuse the first point cloud and the first image. In this way, the computer device divides the first point cloud into multiple sub-point clouds based on the field of view partition of the point cloud acquisition device, divides the first image into multiple sub-images based on the field of view partition, and divides each sub-image into multiple image blocks. One image block among the multiple image blocks obtained by division does not include pixels in two sub-images, thereby avoiding the problem of ghosting of fused data caused by including pixels in two sub-images in one image block, thereby improving the accuracy of the obtained fused point cloud and facilitating navigation and obstacle avoidance of autonomous driving vehicles.

[0007] In a possible implementation, dividing each of the multiple sub-images into multiple image blocks includes: dividing a first sub-image into multiple first image blocks, wherein different first image blocks have the same size; and dividing a second sub-image into multiple second image blocks, wherein different second image blocks have the same size.

[0008] In another possible implementation, dividing the first sub-image into a plurality of first image blocks includes: obtaining a target interval, where the target interval is an interval where the density of the registration points of the first sub-image is located. The registration point is a pixel point in the first image that has a corresponding scanning point in the first point cloud. The first pixel point in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinates of the first scanning point in the coordinate system of the first image are the same as the two-dimensional coordinates of the first pixel point in the coordinate system of the first image. According to the correspondence between each interval in the plurality of intervals and the image block size, the image block size corresponding to the target interval is obtained, and the obtained image block size is used as the first image block size. Based on the first image block size, the first sub-image is divided into a plurality of first image blocks. In this way, for two sub-images, the target intervals where the density of the registration points of the two sub-images obtained by the computer device are located are different, that is, the density of the registration points of the two sub-images is different, and the image block sizes obtained from the correspondence between each interval in the plurality of intervals and the image block size according to the target interval are different, and the image block sizes in the two sub-images obtained by dividing the sub-images based on the obtained image block size are different, and the image block sizes in the same sub-image are the same. The correspondence between each of the multiple intervals and the image block size can be preset in advance based on experience. Therefore, the computer device can select the appropriate image block size according to the density of the registration points to solve the problem of poor accuracy of the fused point cloud.

[0009] In another possible implementation, dividing the first sub-image into a plurality of first image blocks includes: obtaining a first graphic, the first graphic being determined based on the registration points in the first sub-image. Determining the number of first image blocks based on the similarity between the first graphic and the target graphic. Based on the number of first image blocks, dividing the first sub-image into a plurality of first image blocks. In this way, the computer device obtains the similarity between the first graphic and the target graphic, and obtains the number of divided image blocks according to the obtained similarity. The lower the similarity, the more uneven the distribution of the registration points. If the distribution of the registration points is uneven, the registration point density in some areas of the first sub-image is large, and the registration point density in some areas is small. Therefore, the first sub-image needs to be divided into more first image blocks to ensure that the number of registration points in each of the divided first image blocks is relatively small. In this way, the computer device determines the number of divided first image blocks in the first sub-image according to the similarity between the first graphic and the target graphic, thereby determining the size of the first image block, so that the computer device selects a suitable image block size to solve the problem of poor accuracy of the fused point cloud.

[0010] In another possible implementation, each of the multiple sub-images is divided into multiple image blocks, including: extracting a target object in a first sub-image, and dividing the first sub-image into multiple first image blocks based on the extracted target object. For example, the target objects in the first sub-image include images of pedestrians and images of vehicles. If the first sub-image is not divided according to the target objects, then in the image blocks obtained by dividing the first sub-image, it is possible that the same image block contains both pixel points in the image of pedestrians and pixel points in the image of vehicles, which can easily cause ghosting or matching errors in the acquired fused point cloud. If the first sub-image is divided based on the target object, it will be avoided that the pixels in different target objects belong to the same image block, thus improving the accuracy of the acquired fused point cloud.

[0011] In another possible implementation, each of the multiple sub-images is divided into multiple image blocks, including: taking each registration point in the first sub-image as a pixel point in a first image block. Based on the distance between the non-registration point in the first sub-image and each registration point, the non-registration point is assigned to the first image block to which one of the registration points belongs in each registration point. In this way, when dividing the sub-image, taking a registration point in the sub-image as a reference, the non-registration point closest to the registration point can be assigned to the registration point, and the minimum outer bounding rectangular area of ​​the area formed by the non-registration point and the registration point is taken as an image block, thereby limiting the number of registration points included in each image block, and the number of registration points included in the image block determines that the image block size is different when the registration point density is different. For example: when the registration point density is relatively large, a relatively small image block can include 5 registration points, and when the registration point density is relatively small, a large image block can include 5 registration points, so that the sub-image is divided into image blocks of appropriate size, thereby improving the accuracy of the acquired fused point cloud.

[0012] In another possible implementation, the method further includes: obtaining multiple groups of first partition parameters based on the target point cloud; wherein the target point cloud is a point cloud used to describe a second scene in the autonomous driving environment. The second scene is a scene before the first scene. Different groups of first partition parameters are obtained based on different partitioning methods. Obtain the weight of each group of first partition parameters in the multiple groups of first partition parameters. Obtain the second partition parameters based on the multiple groups of first partition parameters and the weight of each group of first partition parameters. Use the second partition parameters to divide the field of view of the point cloud acquisition device into multiple field of view partitions. Wherein, the multiple field of view partitions include the first field of view partition and the second field of view partition. In this way, the division of the field of view of the point cloud acquisition device combines multiple partitioning methods, making the field of view division of the point cloud acquisition device more reasonable. Because the total number of points of the point cloud acquisition device when acquiring a frame of point cloud is certain. The point cloud acquisition device can scan at different scanning densities in different field of view partitions. When the scanning density in one field of view partition increases, the scanning density of the remaining field of view partitions will inevitably decrease to a certain extent. Therefore, reasonable field of view partitioning of the point cloud acquisition device helps to increase the amount of information of the scanning points acquired by the point cloud acquisition device. For example, if many points are placed on the same object in a field of view partition, the information of multiple scanning points in the acquired point cloud represents the object, resulting in a waste of scanning points. If different points are placed on different objects as much as possible in a field of view partition, the information of multiple scanning points in the acquired point cloud represents different objects, thereby increasing the amount of information of the acquired scanning points.

[0013] In another implementation, the method further includes: obtaining a resolution of the first image. If the resolution of the first image is greater than a first threshold, then based on any one of the fused point cloud, the first point cloud or the first image, at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition is adjusted. Wherein, the fused point cloud is a point cloud obtained by fusing the first image and the first point cloud. If the resolution of the first image is less than or equal to the first threshold, then based on the first point cloud or the fused point cloud, at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition is adjusted. In this way, the problem of inaccurate judgment basis caused by the inability of the first image to accurately describe the first scene and adjusting the scanning density of each field of view partition in the laser radar field of view based on the first image can be avoided to adjust the scanning density of each field of view partition.

[0014] In a second aspect, a method for dividing the field of view of a point cloud acquisition device is provided, the method comprising: obtaining multiple groups of first partition parameters and the weight of each group of first partition parameters in the multiple groups of first partition parameters. The first partition parameters are used to partition the field of view of the point cloud acquisition device. Different groups of first partition parameters are obtained based on different partitioning methods. Based on the multiple groups of first partition parameters and the weight of each group of first partition parameters, the second partition parameters are obtained. Using the second partition parameters, the field of view of the point cloud acquisition device is divided into multiple field of view partitions. In this way, the division of the field of view of the point cloud acquisition device combines multiple partitioning methods, making the field of view division of the point cloud acquisition device more reasonable. Because the total number of points of the point cloud acquisition device when acquiring a frame of point cloud is certain. The point cloud acquisition device can scan at different scanning densities in different field of view partitions. When the scanning density in one field of view partition increases, the scanning density of the remaining field of view partitions will inevitably decrease to a certain extent. Therefore, reasonable field of view partitioning of the point cloud acquisition device helps to increase the amount of information of the scanning points acquired by the point cloud acquisition device. For example, if many points are placed on the same object in a field of view partition, the information of multiple scanning points in the acquired point cloud represents the object, resulting in a waste of scanning points. If different points are placed on different objects as much as possible in a field of view partition, the information of multiple scanning points in the acquired point cloud represents different objects, thereby increasing the amount of information of the acquired scanning points.

[0015] In a possible implementation, the partitioning method includes any one of the following: a partitioning method based on field of view safety weight partitioning; a partitioning method based on scanning point density partitioning; a partitioning method based on target graphic similarity.

[0016] In a third aspect, a method for adjusting the scanning density is provided, the method comprising: obtaining the resolution of a first image. If the resolution of the first image is greater than a first threshold, then based on any one of the fused point cloud, the first image or the first point cloud, at least one of the scanning density of the first sub-point cloud and the scanning density of the second sub-point cloud is adjusted. Wherein, the fused point cloud is a point cloud obtained by fusing the first image and the first point cloud. If the resolution of the first image is less than or equal to the first threshold, then based on the first point cloud or the fused point cloud, at least one of the scanning density of the first sub-point cloud and the scanning density of the second sub-point cloud is adjusted. In this way, the problem of inaccurate judgment basis caused by the inability of the first image to accurately describe the first scene and adjusting the scanning density of each field of view partition in the laser radar field of view based on the first image can be avoided, so as to adjust the scanning density of each field of view partition.

[0017] In a possible implementation, adjusting the scanning density of the first sub-point cloud includes: adjusting the scanning density of the first sub-point cloud based on the sub-fused point cloud. The sub-fused point cloud is a sub-fused point cloud obtained by fusing the first sub-point cloud with the first sub-image.

[0018] In a fourth aspect, a point cloud and image fusion device is provided, which can be used to perform any method provided in any possible implementation of the first to third aspects. For example, the fusion device can be a computer device (such as a terminal device, a server or a cloud server) or a chip, etc.

[0019] According to any possible implementation of the first aspect to the third aspect, in a first possible implementation of the ninth aspect, the fusion device may be divided into functional modules according to any method provided in the first aspect to the third aspect. For example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit.

[0020] According to any possible implementation of the first to third aspects, in a second possible implementation of the ninth aspect, the device may include a processor, and the processor is used to execute any method provided by the first to third aspects above.

[0021] In a fifth aspect, a computer-readable storage medium is provided, such as a non-transitory computer-readable storage medium, on which a computer program (or instruction) is stored, and when the computer program (or instruction) is executed on a computer, the computer executes any method provided in any possible implementation of the first to third aspects above.

[0022] According to a sixth aspect, a computer program product is provided, which, when executed on a computer, enables any method provided by any possible implementation of the first to third aspects to be executed.

[0023] In a seventh aspect, a chip is provided, comprising: a processor, configured to call from a memory and run a computer program stored in the memory, and execute any method provided by any possible implementation of the first to third aspects.

[0024] In an eighth aspect, a chip is provided, comprising: a processor, configured to call from a memory and run a computer program stored in the memory, and execute any method provided by any possible implementation of the first to third aspects.

[0025] It can be understood that any of the fusion devices, computer storage media, computer program products or chips provided above can be applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram of the current fusion point cloud;

[0027] Figure 2 is a schematic diagram of the structure of an environment detection system applicable to an embodiment of the present application;

[0028] Figure 3 A schematic diagram of the structure of a computer device to which the technical solution provided in the embodiment of the present application is applicable;

[0029] Figure 4 A schematic diagram of a flow chart of a method for fusing point cloud and image provided in an embodiment of the present application;

[0030] Figure 5 This is an example image of the first point cloud;

[0031] Figure 6 is an example image of the first image;

[0032] Figure 7 A schematic diagram of a flow chart of a method for dividing a laser radar field of view provided in an embodiment of the present application;

[0033] Figure 8 A schematic diagram of a result after a computer device partitions a target point cloud according to a first partition parameter;

[0034] Fig. 9 A flowchart of a method for adjusting the scanning density of different field of view partitions in a laser radar field of view provided in an embodiment of the present application;

[0035] Fig.10 A schematic diagram of the structure of a point cloud and image fusion device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] like Figure 2 , which is a schematic diagram of the structure of an environment detection system applicable to an embodiment of the present application, and the environment detection system includes a computer device 10, a point cloud acquisition device 11, and a camera 12. It should be noted that the embodiment of the present application does not limit the number of computer devices 10, point cloud acquisition devices 11, and cameras 12 in the environment detection system. Figure 2 In the example, the environment detection system includes a computer device 10, a point cloud acquisition device 11 and a camera 12. In the environment detection system, the computer device 10 is connected to the point cloud acquisition device 11 and the camera 12 respectively.

[0037] The computer device 10 is used to send instructions to the point cloud acquisition device 11, so that the point cloud acquisition device 11 partitions the field of view of the point cloud acquisition device according to the instructions, and the point cloud acquisition device 11 scans different field of view partitions with different scanning densities to obtain a point cloud. The computer device 10 is also used to send instructions to the camera 12 to instruct the camera 12 to acquire an image. The computer device 10 receives the point cloud sent by the point cloud acquisition device 11 and the image sent by the camera 12, and fuses the point cloud and the image to obtain a fused point cloud.

[0038] The point cloud acquisition device 11 is used to scan the scene within the field of view of the point cloud acquisition device to obtain a point cloud.

[0039] The camera 12 is used to capture the scene within the camera's field of view to obtain an image.

[0040] like Figure 3 FIG. 1 is a schematic diagram of a computer device to which the technical solution provided in the embodiment of the present application is applicable. In one example, from the perspective of hardware structure, Figure 2 The computer device 10 may be Figure 3 The computer device 10 in. Figure 3 The computer device 10 shown may include at least one processor 101 , a communication line 102 , a memory 103 and at least one communication interface 104 .

[0041] The processor 101 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0042] The communication line 102 may include at least one path, such as a data bus, and / or a control bus, for transmitting information between the above components (such as at least one processor 101, the communication line 102, the memory 103 and at least one communication interface 104).

[0043] The communication interface 104 uses any transceiver or other device for communicating with other devices or communication networks, such as wide area networks (WAN), local area networks (LAN), etc.

[0044] The memory 103 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 103 may be independent and connected to the processor 101 via the communication line 102. The memory 103 may also be integrated with the processor 101. The memory 103 provided in the embodiment of the present application generally includes a non-volatile memory. The memory 103 is used to store computer instructions for executing the solution of the present application, and the execution is controlled by the processor 101. The processor 101 is used to execute the computer instructions stored in the memory 103, thereby implementing the method provided in the following embodiments of the present application.

[0045] The storage 103 includes a memory and a hard disk.

[0046] Optionally, the computer instructions in the embodiments of the present application may also be referred to as application code or system, which is not specifically limited in the embodiments of the present application.

[0047] In a specific implementation, as an embodiment, the computer device 10 may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0048] In a specific implementation, as an embodiment, the computer device 10 may further include an output device 105 and / or an input device 106. The output device 105 communicates with the processor 101 and may display information in a variety of ways. For example, the output device 105 may be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector. The input device 106 communicates with the processor 101 and may receive user input in a variety of ways. For example, the input device 106 may be a mouse, a keyboard, a touch screen device, or a sensor device.

[0049] It should be noted that Figure 3 The computer device shown is only an example and does not limit the computer devices applicable to the embodiments of the present application. In actual implementation, the computer device may include Figure 3 More or fewer devices or components than those shown.

[0050] The following is an explanation of some of the terms used in this application:

[0051] 1) Scene

[0052] A scene refers to an environment within a specific time and space. The environment includes: road conditions, regions, weather, lighting, and traffic participants. Among them, road conditions include: intersections (such as crossroads, T-shaped intersections, or Y-shaped intersections, etc.) and road types (such as expressways, ordinary roads, or mountain roads, etc.). Regions include: urban areas, towns, and mountainous areas. Weather includes: rain, snow, and fog, etc. Lighting includes: day and night, etc. Traffic participants include: moving objects (such as: vehicles, people, and animals, etc.) and stationary objects (such as: trees, telephone poles, street lights, traffic signs, lane lines, stones, trash cans, buildings, and mountains, etc.).

[0053] 2) Point cloud, point cloud acquisition equipment, laser radar, scanning point, scanning density, scanning point density

[0054] A point cloud is a collection of scanning points obtained after a point cloud acquisition device acquires multiple scanning points on the surface of an object. A scanning point has a three-dimensional coordinate, angle, laser reflection intensity, etc.

[0055] Point cloud acquisition equipment includes: 3D laser scanner (e.g., laser radar, such as point scanning laser radar, line scanning laser radar or area array laser radar, etc.), photographic scanner, etc. For the convenience of explanation, the following description is based on the example that the point cloud acquisition equipment is a laser radar.

[0056] Laser radar is a distance measurement system. The distance measurement system includes a light source and a detector, a timer and a processor. The method for laser radar to obtain a scanning point includes: the light source (such as a laser) emits a beam of light (such as a laser) to the target to be measured, and the detector receives the light reflected by the target to be measured. The timer (such as a system clock) calculates the difference between the time point when the light source emits the light and the time point when the detector receives the reflected light. The processor determines the distance between the target to be measured and the device where the distance measurement system is located based on the difference and the speed of light, thereby obtaining a scanning point. The attributes of the scanning point include the three-dimensional coordinates of the scanning point in the field of view of the laser radar (referred to as the three-dimensional coordinates of the scanning point), and the three-dimensional coordinates of the scanning point include the depth of the scanning point. The depth of the scanning point is the distance between the target to be measured and the device where the distance measurement system is located. The laser radar scans the laser radar field of view by the above method to obtain multiple scanning points in the laser radar field of view, and the set composed of the multiple scanning points is a point cloud.

[0057] Scanning density is the number of scanning points acquired by the LiDAR per unit area under ideal conditions. The ideal condition means that the number of scanning points acquired by the LiDAR per unit area is the same as the number of points placed per unit area.

[0058] Scan point density is the number of scan points per unit area in the point cloud.

[0059] 3) Camera

[0060] The camera can be any camera used to collect images. The camera in the embodiment of the present application includes a color (redgreenblue, RGB) camera and a grayscale camera.

[0061] Color cameras use the three-primary color model, which is a color standard in the industry. It is a color model that obtains multiple colors by changing the three color channels of red, green, and blue and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue, where R represents red, G represents green, and B represents blue. This color model includes all the colors that human vision can perceive and is one of the most widely used color models.

[0062] 4) Field of view, LiDAR field of view, camera field of view

[0063] Field of view, usually expressed in degrees, is used to indicate range. Generally, the larger the field of view, the greater the range. The field of view can be 50 degrees, 58 degrees, 52 degrees, 180 degrees, or 360 degrees, etc.

[0064] The laser radar field of view refers to the range that the laser radar can detect. The embodiment of the present application does not limit the size of the laser radar field of view.

[0065] The camera field of view refers to the range that the camera can capture. The embodiment of the present application does not limit the size of the camera field of view.

[0066] 5) Object points, image points, points of the same name, feature points

[0067] A point in a scene is called an object point.

[0068] A point in an image is called an image point. An image point in the image of a scene corresponds to an object point in the scene.

[0069] The same-name points are image points (such as scanning points in the first point cloud and pixel points in the first image) of an object point in the scene in different images (such as the first point cloud and the first image in the following text).

[0070] A feature point is a pixel or scanning point in an image that is stable, rotationally invariant, and can overcome grayscale inversion.

[0071] 6) Target audience

[0072] The target object is an image representing a person or object in the scene, for example, an image of a person, an image of a car, an image of a road sign, an image of a bicycle, an image of a dog, etc.

[0073] 7) Other terms

[0074] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0075] In the embodiments of the present application, "at least one" means one or more, and "plurality" means two or more.

[0076] In the embodiments of the present application, "and / or" is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] like Figure 4 FIG. 1 is a flow chart of a method for fusing point cloud and image provided in an embodiment of the present application. For example, the present embodiment can be applied to Figure 2 The environmental detection system shown in Figure 2 The point cloud acquisition device in the figure is taken as a laser radar as an example. Figure 4 The method shown may include the following steps:

[0079] S100: The laser radar acquires a first point cloud. The first point cloud is a point cloud of a first scene in an autonomous driving environment. The first point cloud includes a first sub-point cloud and a second sub-point cloud. The first sub-point cloud is used to describe a first sub-scene in the first scene. The second sub-point cloud is used to describe a second sub-scene in the first scene.

[0080] Specifically, the laser radar scans the first scene in the autonomous driving environment to obtain a first point cloud. The first scene is divided into a plurality of different sub-scenes according to different field of view partitions, and the field of view partitions correspond to the sub-scenes one by one. The plurality of different sub-scenes include a first sub-scene and a second sub-scene. The laser radar scans the first sub-scene to obtain a first sub-point cloud, and scans the second sub-scene to obtain a second sub-point cloud. The embodiment of the present application does not limit the method for dividing the field of view of the laser radar.

[0081] For example, the first point cloud is Figure 5 shown.

[0082] S101: The laser radar sends a first point cloud to a computer device.

[0083] S102: The camera acquires a first image, where the first image is an image of a first scene, that is, an image obtained by the camera shooting the first scene.

[0084] For example, the first image is Figure 6 shown.

[0085] S103: The camera sends the first image to the computer device.

[0086] It should be noted that the embodiment of the present application does not limit the execution order of S100-S101 and S102-S103. For example, S102-S103 is executed before S100-S101.

[0087] S104: The computer device divides the first image into a plurality of sub-images. The plurality of sub-images include a first sub-image and a second sub-image. The first sub-image is used to describe the first sub-scene, and the second sub-image is used to describe the second sub-scene.

[0088] Specifically, the computer device obtains the two-dimensional coordinates in the coordinate system of the first image corresponding to the three-dimensional coordinates of the boundaries of each field of view partition of the laser radar field of view based on the mapping relationship between the three-dimensional coordinates of the scanning points in the coordinate system of the first point cloud and the two-dimensional coordinates of the pixel points in the coordinate system of the first image. Then, the computer device divides the first image into a plurality of sub-images based on the two-dimensional coordinates in the coordinate system of the first image corresponding to the three-dimensional coordinates of the boundaries of each field of view partition of the laser radar field of view. The sub-images correspond to the field of view partitions one by one.

[0089] It should be noted that the embodiment of the present application does not limit the method for obtaining the mapping relationship between the three-dimensional coordinates in the coordinate system where the first point cloud is located and the two-dimensional coordinates in the coordinate system where the first image is located.

[0090] In one implementation, the computer device obtains the mapping relationship between the three-dimensional coordinates of the scanning points in the coordinate system where the first point cloud is located and the two-dimensional coordinates of the pixel points in the coordinate system where the first image is located through direct linear transformation (DLT).

[0091] In this implementation, the mapping relationship between the three-dimensional coordinates of the scanning point in the coordinate system of the first point cloud and the two-dimensional coordinates of the pixel point in the coordinate system of the first image can be expressed by the following formula:

[0092]

[0093] Among them, (x w ,y w ,z w ) is the coordinate of the wth scanning point in the first point cloud, where w is a positive integer. (u,v) is the coordinate of the pixel point corresponding to the wth scanning point in the coordinate system of the first image. l1 to l11 are 11 undetermined parameters of the direct linear transformation method, which characterize the mapping relationship between the three-dimensional coordinates of the scanning point in the coordinate system of the first point cloud and the two-dimensional coordinates of the pixel point in the coordinate system of the first image.

[0094] In another implementation, the computer device obtains the correspondence between the three-dimensional coordinates in the coordinate system where the first point cloud is located and the two-dimensional coordinates in the coordinate system where the first image is located through beam adjustment.

[0095] In this implementation, the mapping relationship between the three-dimensional coordinates of the scanning point in the coordinate system of the first point cloud and the two-dimensional coordinates of the pixel point in the coordinate system of the first image can be expressed by the following formula:

[0096] (u,v)=P(x w ,y w ,zw ), where P is a transformation matrix. The transformation matrix P represents the mapping relationship between the three-dimensional coordinates of the scanning points in the coordinate system of the first point cloud and the two-dimensional coordinates of the pixel points in the coordinate system of the first image.

[0097] In the above two implementations, direct linear transformation requires computer equipment to solve l1 to l 11 The values ​​of these 11 undetermined parameters require the computer to solve the transformation matrix P. The computer obtains the same-name points in the first point cloud and the first image, and substitutes the coordinates of the scanned points in the first point cloud in the same-name points into (x w ,y w ,z w ), substitute the coordinates of the pixel in the first image in the same name point into (u,v) to solve the 11 unknown parameters in the direct linear transformation or the transformation matrix in the bundle adjustment.

[0098] The method for a computer device to obtain points with the same name in a first point cloud and a first image may include:

[0099] Step 1: The computer device establishes a plane coordinate system of the first point cloud, which is the same as the coordinate system of the first image.

[0100] Step 2: The computer device projects the first point cloud into a first mapping image in the established plane coordinate system.

[0101] Specifically, the computer device may use any one of perspective projection, parallel projection (a method of projecting each point on the first point cloud vertically onto the plane of the first mapping image using the plane of the first image as the plane of the first mapping image) or scanning projection (a method of projecting according to the point cloud scanning process based on the principle of point cloud acquisition) to map each scanned point in the first point cloud to the coordinate system of the first image to obtain the first mapping image. A scanned point in the first point cloud corresponds to a pixel point in the first mapping image.

[0102] Step 3: The computer device uses a feature point extraction algorithm to obtain feature points in the first mapping image and feature points in the first image based on the first mapping image and the first image. The feature point is a stable, rotationally invariant, effective feature that can overcome grayscale inversion. The feature point extraction algorithm can be a Moravec operator, a Forstner operator, a Harris operator, etc.

[0103] Step 4: The computer device uses a feature point matching algorithm to establish a correspondence between the feature points in the first mapping image and the feature points in the first image. The feature point matching algorithm may be a correlation coefficient method, a relaxation method, or a least squares method.

[0104] Step 5: The computer device obtains the relationship between the scanning point in the first point cloud and the pixel point in the first image based on the correspondence between the feature points in the first mapping image and the feature points in the first image, and the correspondence between the pixel points in the first mapping image and the scanning point in the first point cloud. The scanning point in the first point cloud and the pixel point in the first image that have a corresponding relationship are a pair of points with the same name.

[0105] The computer device obtains 11 undetermined parameters in direct linear transformation or a transformation matrix in bundle adjustment based on the obtained multiple pairs of same-name points, thereby obtaining the correspondence between the three-dimensional coordinates in the coordinate system of the first point cloud and the two-dimensional coordinates in the coordinate system of the first image.

[0106] In this embodiment, the point cloud obtained by laser radar scanning and the image taken by the camera are taken as examples to illustrate the method of fusing the point cloud and the image. Optionally, the computer device may first process the point cloud obtained by laser radar scanning and / or the image taken by the camera, and then fuse the processed point cloud and / or image.

[0107] S105: The computer device divides each sub-image in the first image into a plurality of image blocks.

[0108] Specifically, the method for dividing the first sub-image by a computer device may include the following method, wherein the first sub-image may be any sub-image in the first image:

[0109] Method 1: The computer device divides the first sub-image into a plurality of first image blocks, wherein the sizes of different first image blocks are the same.

[0110] In one implementation of method 1:

[0111] First, the computer device obtains the target interval. The target interval is the interval where the density of the registration points of the first sub-image is located. The registration point is a pixel point in the first image that has a corresponding scanning point in the first point cloud; the first pixel point in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinates of the first scanning point in the coordinate system of the first image are the same as the two-dimensional coordinates of the first pixel point in the coordinate system of the first image.

[0112] Then, the computer device obtains the image block size corresponding to the target interval according to the correspondence between each interval in the multiple intervals and the image block size, and uses the obtained image block size as the first image block size. The correspondence between each interval in the multiple intervals and the image block size can be predefined in the computer device. The correspondence between each interval in the multiple intervals and the image block size can be set based on experience.

[0113] Next, the computer device divides the first sub-image into a plurality of first image blocks based on the first image block size.

[0114] In this way, the target intervals obtained by the computer device are different, that is, the registration point density of the first sub-image is different, and the image block sizes of the divided first sub-image are different. Therefore, the computer device can select a suitable image block size according to the registration point density to solve the problem of poor accuracy of the fused point cloud.

[0115] Another implementation of method 1:

[0116] First, the computer device obtains a first graphic, wherein the first graphic is determined based on the registration points in the first sub-image. For example, the first graphic is a graphic formed by the registration points in the first sub-image.

[0117] Secondly, the computer device determines the number of the first image blocks based on the similarity between the first figure and the target figure. As an example, the target figure includes any one of a rectangle, a parallelogram or a trapezoid.

[0118] Next, the computer device divides the first sub-image into a plurality of first image blocks based on the number of first image blocks.

[0119] In this method, the computer device obtains the degree of similarity between the first graphic and the target graphic, and obtains the number of divided image blocks according to the obtained degree of similarity. The lower the degree of similarity, the more uneven the distribution of the registration points. If the distribution of the registration points is uneven, the registration point density in some areas of the first sub-image is large, and the registration point density in some areas is small. Therefore, more first image blocks need to be divided into the first sub-image to ensure that the number of registration points in each of the divided first image blocks is less than the first threshold. In this way, the computer device determines the number of first image blocks divided in the first sub-image according to the degree of similarity between the first graphic and the target graphic, thereby determining the size of the first image block, so that the computer device selects a suitable image block size to solve the problem of poor accuracy of the fused point cloud.

[0120] Method 2: A computer device extracts a target object from a first sub-image, and divides the first sub-image into a plurality of first image blocks based on the extracted target object.

[0121] In one implementation of the second method, the computer device extracts the target object in the first sub-image, and uses the minimum enclosing rectangle of each target object as a first image block. Based on this, the computer device can divide the first sub-image into a plurality of first image blocks. The target object includes at least one target object in an image database. The image database may include: Pascal Visual Object Classes, Microsoft Common Object in Context, or City Scapes.

[0122] Method three: Based on the texture features of the first sub-image, the computer device uses the smallest enclosing rectangular area of ​​the continuous area with the same texture features in the first sub-image as a first image block. Based on this, the computer device can divide the first sub-image into multiple first image blocks.

[0123] Method 4: The computer device regards each registration point in the first sub-image as a pixel point in the first image block. Then, based on the distance between the non-registration point in the first sub-image and each registration point, the computer device assigns the non-registration point to the first image block to which one of the registration points belongs.

[0124] In one implementation of method 4, the computer device assigns the pixel points of the non-registration points in the first sub-image to the registration points in the first sub-image that are closest to the non-registration points. Based on this, the computer device obtains the minimum enclosing rectangular area of ​​the figure composed of each registration point and the non-registration points assigned to the registration point, and uses the minimum enclosing rectangular area as a first image block.

[0125] In the embodiment of the present application, the method by which the computer device divides the sub-image into image blocks may also be other methods in the prior art, such as any one or more combinations of a threshold-based segmentation method, a sub-image-based segmentation method, an edge-based segmentation method, an image segmentation method based on cluster analysis, a segmentation method based on fuzzy set theory, or a segmentation method based on gene coding.

[0126] S106: The computer device fuses the first point cloud and the first image based on the multiple image blocks.

[0127] Specifically, the computer device determines the scanning point corresponding to the registration point in the first point cloud according to the registration point in each image block of the multiple image blocks, and fuses the scanning point with the image block where the registration point corresponding to the scanning point is located.

[0128] It should be noted that, for image blocks that do not include registration points, the computer device can merge the image blocks that do not include registration points and the image blocks that include registration points into one image block based on the distance between the image blocks, and fuse the image block with the scanning points in the point cloud corresponding to the registration points in the image blocks.

[0129] The embodiment of the present application does not limit the method for fusing scanning points with image blocks. Exemplarily, a computer device uses the color attributes of the image block (such as RGB values) and the attributes of the scanning points corresponding to the image block (such as three-dimensional coordinates, angles, and laser reflection intensity, etc.) to generate a fused point cloud, and the fused point cloud has both the attributes of the scanning points and the attributes of the image block.

[0130] Subsequently, the computer equipment can perform navigation and obstacle avoidance for the autonomous vehicle based on the fused point cloud.

[0131] In the embodiment of the present application, the field of view partition of the laser radar divides the first point cloud into a plurality of sub-point clouds, and the computer device divides the first image into a plurality of different sub-images based on the field of view partition of the laser radar. The computer device divides each sub-image into a plurality of image blocks, and fuses the first point cloud and the first image based on the plurality of image blocks. In this way, one of the plurality of image blocks obtained by the computer device will not include pixels in two sub-images, thereby avoiding the problem of ghosting of fused data caused by including pixels in two sub-images in one image block, thereby improving the accuracy of the obtained fused point cloud, and facilitating the navigation and obstacle avoidance of the autonomous driving vehicle.

[0132] like Figure 7 FIG. 1 is a flow chart of a method for dividing a laser radar field of view provided in an embodiment of the present application. For example, this embodiment can be applied to Figure 2 The environmental detection system shown in Figure 2 The point cloud acquisition device in the figure is taken as a laser radar as an example. Figure 7 The method shown may include the following steps:

[0133] S200: The laser radar scans the current scene in the autonomous driving environment within the laser radar field of view to obtain a target point cloud. The laser radar field of view is divided into a plurality of different field of view partitions according to the second partition parameter 1. The second partition parameter 1 includes indication information of a plurality of field of view partitions for dividing the laser radar field of view. The indication information of the field of view partitions may be identified by the boundary coordinates of the field of view partitions; or, the indication information of the field of view partitions may be identified by the division ratio of the laser radar field of view. For example, the horizontal division ratio of the laser radar field of view is 1:2:1, and the vertical division ratio is 1:8:1.

[0134] S201: The laser radar sends a target point cloud to a computer device.

[0135] S202: The computer device obtains multiple groups of first partition parameters based on the target point cloud. One group of first partition parameters is used to partition the target point cloud. Different groups of first partition parameters are obtained based on different partitioning methods. The weights of the first partition parameters obtained by the computer device based on different partitioning methods are different. The sum of the weights of the multiple groups of first partition parameters is 1.

[0136] Specifically, the computer device may obtain multiple groups of first partition parameters based on the target point cloud in the following manner.

[0137] Method 1: The computer device obtains a set of first partition parameters based on the field of view safety ratio.

[0138] Specifically, the computer device obtains the field of view safety ratio corresponding to the scene of the target point cloud from the corresponding relationship between each scene in multiple scenes and the field of view safety ratio according to the scene of the target point cloud. Among them, the field of view safety ratio is reflected by a set of first partition parameters. The corresponding relationship between each scene in multiple scenes and the field of view safety ratio can be preset based on experience. Exemplarily, the scene of the target point cloud is a high-speed scene, and the computer device obtains the field of view safety ratio corresponding to the high-speed scene from the corresponding relationship between each scene in multiple scenes and the field of view safety ratio. The field of view safety ratio is a 1:8:1 ratio for the longitudinal division of the laser radar field of view. Because the safety ratio in front is relatively high when the vehicle is driving at high speed, the field of view safety ratio obtained by the computer device is a 1:8:1 ratio for the longitudinal division of the laser radar field of view.

[0139] In a second method, the computer device obtains a set of first partition parameters based on the scanning point density of the target point cloud.

[0140] In one implementation, the computer device determines any scanning point in the target point cloud as a growth point and grows a same region in different directions to obtain multiple regions; if the number of scanning points in each of the multiple regions is approximately equal, it means that the scanning point density in the multiple regions is approximately the same. Based on this, the computer device grows a continuous range with approximately the same scanning point density into a rectangular region, and obtains the boundary coordinates of each rectangular region to obtain a set of first partition parameters.

[0141] Exemplarily, a set of first partitioning parameters acquired by the computer device is: {P1, (25*5); P2, (25*15); P3, (25*5)}, wherein P1 includes the coordinates of the vertices of the first area, 25 is the length of the divided first area, and 5 is the width of the divided first area. P2 includes the coordinates of the vertices of the second area, 25 is the length of the divided second area, and 15 is the width of the divided second area. P3 includes the coordinates of the vertices of the third area, 25 is the length of the divided third area, and 5 is the width of the divided third area. The first area, the second area, and the third area all represent regular rectangular partitions, and P1 can be a set of four vertex coordinates of the rectangular partition. The result of the computer device partitioning the target point cloud according to the set of first partitioning parameters is as follows. Figure 8 shown.

[0142] In a third method, the computer device obtains a set of first partition parameters based on the target graph similarity.

[0143] Specifically: First, the computer device projects the target point cloud into a mapping image in a plane coordinate system perpendicular to the horizontal plane. Then, based on the mapped image, the computer device obtains the similarity between the geometric figure formed by the same-name points of the scanning points in the target point cloud in the mapped image and the target figure, and uses the boundary pixel points of the geometric figure with a similarity greater than a threshold as growth points, and uses the region generation method to obtain multiple regular rectangular regions. The target figure includes: any one of a rectangle, a parallelogram or a trapezoid. Finally, the computer device obtains the boundary coordinates of multiple regular rectangular regions to obtain a set of first partition parameters.

[0144] Of course, the computer device may also use other methods in the prior art to obtain a set of first partition parameters, which is not limited in the embodiments of the present application.

[0145] S203: The computer device obtains second partition parameter 2 based on the obtained multiple groups of first partition parameters and the weight of each group of first partition parameters in the multiple groups of first partition parameters. The second partition parameter 2 is used to divide the laser radar field of view into multiple field of view partitions.

[0146] Specifically, the computer device calculates the second partition parameter 2 based on the obtained multiple groups of first partition parameters and the weight of each group of first partition parameters.

[0147] Exemplarily, the computer device obtains three groups of first partition parameters. The first group of first partition parameters is that the laser radar field of view is divided into a horizontal ratio of 1:2:1, and the weight of the first group of first partition parameters is 20%. The second group of first partition parameters is that the laser radar field of view is divided into a horizontal ratio of 1:1:1:1, and the weight of the second group of first partition parameters is 50%. The third group of first partition parameters is that the laser radar field of view is divided into a horizontal ratio of 1:1.5:1, the weight of the third group of first partition parameters is 30%, and the laser radar field of view is divided into a vertical ratio of 1:8:1, and the weight is 100%. The computer device calculates a group of second partition parameters according to the three groups of first partition parameters and the weights of each group of first partition parameters, and the group of second partition parameters includes the laser radar field of view horizontal division ratio and the laser radar field of view vertical division ratio. Among them, the horizontal division ratio of the laser radar field of view is: (1*20%+1*50%+1*30%): (2*20%+1*50%+1.5*30%): (1*20%+1*50%+1*30%): (0+1*50%+0)=1:1.35:1:0.5. According to the same calculation method, the vertical division ratio of the laser radar field of view is 1:8:1.

[0148] It is understandable that during the driving process of the autonomous driving vehicle, the environment detection system can obtain multiple sets of second partition parameters through the above S200~S203. Exemplarily, the computer device obtains a point cloud of a scene every 5ms, and obtains and stores a set of second partition parameters through the above S200~S203. After each set of second partition parameters is obtained, the computer device can perform the following steps to obtain the second partition parameter 2.

[0149] Step 1: The computer device obtains the number of groups of stored second partition parameters.

[0150] Exemplarily, the computer device stores the acquired set of second partition parameters in a data structure (eg, a table, an array, a linked list, a queue, or a stack), and the computer device acquires the number of sets of the second partition parameters stored in the data structure.

[0151] Step 2: The computer device determines whether the number of groups of the first partition parameter is greater than a second threshold value. The second threshold value can be determined according to the number of point cloud frames required for the computer device to determine that the scene of the point cloud acquired by the laser radar is stable.

[0152] If yes, it means that the number of groups of the second partition parameter reaches the minimum standard, and step 3 is performed. If no, it means that the number of groups of the second partition parameter does not reach the minimum standard, and the process ends.

[0153] Step 3: The computer device obtains the stored fluctuation values ​​of multiple sets of second partition parameters.

[0154] For example, the computer device calculates at least one of the variance, standard deviation or covariance of the multiple groups of second partition parameters, and uses the calculated variance, standard deviation or covariance as the fluctuation value of the multiple groups of second partition parameters.

[0155] Step 4: The computer device determines whether the fluctuation values ​​of the multiple sets of second partition parameters are greater than the third threshold value. If not, it means that the laser radar field of view division is stable, and step 5 is executed. If yes, it means that the laser radar field of view division is unstable, and the process ends. The third threshold value can be determined according to the accuracy required for changing the laser radar field of view division.

[0156] Step 5: The computer device obtains the second partition parameter 2. The second partition parameter 2 is any one of the stored multiple sets of second partition parameters.

[0157] The computer device preferentially selects the most recently stored second partition parameter as the second partition parameter 2 .

[0158] Step 6: The computer device determines that the second partition parameter 1 is different from the second partition parameter 2.

[0159] It should be noted that if the computer device determines that the second partition parameter 1 is the same as the second partition parameter 2, the process ends.

[0160] After executing step 6, the computer device may delete the stored multiple sets of second partition parameters.

[0161] S204: The computer device sends instruction information to the laser radar, where the instruction information is used to instruct the laser radar to divide the laser radar field of view according to the second partition parameter 2 in the instruction information.

[0162] It should be noted that this embodiment can be executed before the first embodiment. In this case, the target point cloud is a frame of point cloud acquired before the first point cloud, which is used to partition the laser radar field of view for acquiring the first point cloud in the first embodiment. This embodiment can also be executed after the first embodiment. In this case, the target point cloud can be the first point cloud, which is used to partition the laser radar field of view for acquiring the second point cloud. The second point cloud is a frame of point cloud acquired after the first point cloud.

[0163] S205: The laser radar divides the laser radar field of view according to the instruction information.

[0164] In this embodiment, the computer device obtains multiple groups of first partition parameters according to the target point cloud, and obtains the weight of each group of partition parameters in the multiple groups of first partition parameters. The partitioning methods of any two different groups of first partition parameters are different. Then, the computer device calculates the second partition parameter 2 according to the multiple groups of first partition parameters and the weight of each group of first partition parameters. Finally, the laser radar field of view is divided using the second partition parameter 2. In this way, the division of the laser radar field of view combines multiple partitioning methods, making the laser radar field of view division more reasonable. Because the total number of points when the laser radar obtains a frame of point cloud is certain. The laser radar can scan at different scanning densities in different field of view partitions. When the scanning density in a field of view partition increases, the scanning density of the remaining field of view partitions will inevitably decrease to a certain extent. Therefore, reasonable laser radar field of view partitioning helps to increase the amount of information of the scanning points obtained by the laser radar. Exemplarily, if many points are punched on the same object under a field of view partition, the information of multiple scanning points in the acquired point cloud represents the object, resulting in a waste of scanning points. If different points are placed on different objects as much as possible in one field of view partition, the information of multiple scanning points in the acquired point cloud represents different objects, thereby increasing the amount of information of the acquired scanning points.

[0165] like Fig. 9 FIG. 1 is a flow chart of a method for adjusting the scanning density of different field of view partitions in the laser radar field of view provided by an embodiment of the present application. For example, this embodiment can be applied to Figure 3 Computer equipment shown. Fig. 9 The method shown may include the following steps:

[0166] S300: The computer device acquires a first fused point cloud. The first fused point cloud is a fused point cloud of a first point cloud and a first image. The first point cloud is a point cloud of a first scene in an autonomous driving environment, and the first image is an image of the first scene. The first point cloud includes a first sub-point cloud. The first sub-point cloud and the first sub-image are used to describe a first sub-scene in the first scene.

[0167] Specifically, the first fused point cloud is obtained by referring to the method in the first embodiment, which will not be described in detail. The laser radar field of view includes a first field of view partition. The laser radar scans the first sub-scene at a first scanning density in the first field of view partition to obtain a first sub-point cloud.

[0168] S301: The computer device determines whether the resolution of the first image is greater than a fourth threshold. If so, it means that the first image can more accurately describe the first scene, and then S302 is executed. If not, it means that the first image cannot accurately describe the first scene, and then S303 is executed.

[0169] S302: The computer device adjusts the scanning density of each of the multiple field of view partitions in the current laser radar field of view based on the first fused point cloud, the first image or any one of the first point cloud.

[0170] The embodiment of the present application does not limit the implementation method of adjusting the scanning density of each field of view partition in the laser radar field of view.

[0171] In one implementation, the computer device adjusts the first scanning density based on the characteristics of the first sub-fused point cloud, wherein the first sub-fused point cloud is a fused point cloud obtained by fusing the first sub-point cloud with the first sub-image.

[0172] In the first example, the computer device obtains the area of ​​the region of the texture feature with the largest area of ​​the same texture feature in the first sub-fusion point cloud. If the ratio of the area of ​​the obtained region to the area of ​​the first sub-fusion point cloud is greater than the fifth threshold, it means that only a small number of scanning points are needed to describe the first sub-scene, and the first scanning density is reduced. If the ratio of the area of ​​the obtained region to the area of ​​the first sub-fusion point cloud is less than or equal to the fifth threshold, it means that more scanning points are needed to describe the first sub-scene, and the first scanning density is increased.

[0173] In the second example, the computer device uses a feature point selection algorithm to obtain multiple feature points in the first sub-fusion point cloud. The computer device calculates the difference value of the depth of the multiple feature points obtained (such as variance, standard deviation or covariance, etc.). Among them, the depth of a feature point represents the distance between the traffic participant in the real scene described by the feature point and the laser radar. If the depth difference value calculated by the computer device is greater than the sixth threshold, it means that more scanning points are needed to describe the first sub-scene, and the first scanning density is increased; if the depth difference value calculated by the computer device is less than or equal to the sixth threshold, it means that the current scanning point can describe the first sub-scene, and the first scanning density can be maintained unchanged.

[0174] In the third example, the computer device determines whether the number of scanning points included in the first sub-fused point cloud area under the first field of view partition is less than the seventh threshold. If the number of scanning points included is less than the seventh threshold, and the depth of the scanning point with the smallest depth included is greater than the eighth threshold, it means that there is no object within a certain distance range from the laser radar under the first field of view partition (for example, the actual scene under the first field of view partition is mostly sky, and a small part is street lights in the distance). At this time, the computer device uses the preset minimum scanning density to ensure safety requirements as the first scanning density.

[0175] In the fourth example, the computer device obtains the number of target objects in the first sub-fusion point cloud. The target objects include at least one target in the image database. If the number of target objects obtained by the computer device is greater than the ninth threshold, the first scanning density is increased; if the number of targets obtained by the computer device is less than or equal to the ninth threshold, the current first scanning density is maintained unchanged.

[0176] It should be noted that in the above implementation, if the information determined by the computer device can be obtained from the point cloud, the scanning density of each field of view partition can also be adjusted based on the obtained point cloud alone (such as the second example, the third example and the fourth example above). If the information determined by the computer device can be obtained from the image, the computer device can also adjust the scanning density of each field of view partition based on the image alone (such as the first example and the fourth example above).

[0177] S303: The computer device adjusts the scanning density of each of the multiple field of view partitions in the current lidar field of view based on the first point cloud or the first fused point cloud.

[0178] For example, refer to the second example, the third example, and the fourth example in step S302.

[0179] The computer device can send instruction information to the laser radar, and the instruction information is used to instruct the laser radar to adjust the scanning density of the field of view partition. The laser radar can adjust the scanning density by controlling the time interval of the laser dotting.

[0180] In this embodiment, the computer device can determine whether to adjust the scanning density of each field of view partition in the laser radar field of view based on the first image according to the resolution of the first image. If the resolution of the first image is not judged, and the first image cannot accurately describe the first scene, and the scanning density of each field of view partition in the laser radar field of view is adjusted based on the first image, since the first image cannot accurately describe the first scene, the judgment basis for adjusting the scanning density of each field of view partition in the laser radar field of view obtained by the computer device is inaccurate. The embodiment of the present application can avoid the problem of inaccurate judgment basis caused by this, so that the computer device can better control the scanning density of each field of view partition.

[0181] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0182] The embodiment of the present application can divide the functional modules of the computer device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0183] like Fig.10 As shown, it is a schematic diagram of the structure of a point cloud and image fusion device provided in an embodiment of the present application. The point cloud and image fusion device 20 can be used to execute any of the above embodiments (such as Figure 4 , Figure 7 or Fig. 9 The function performed by the computer device in the embodiment shown in the figure). The point cloud and image fusion device 20 includes: an acquisition module 201, a division module 202 and a fusion module 203. Optionally, the point cloud and image fusion device 20 also includes an adjustment module 204. The acquisition module 201 is used to acquire a first point cloud. The first point cloud is a point cloud used to describe a first scene in an autonomous driving environment. The first point cloud includes a first sub-point cloud and a second sub-point cloud, and the first sub-point cloud and the second sub-point cloud are used to describe a first sub-scene and a second sub-scene in the first scene, respectively. The first sub-scene and the second sub-scene are sub-scenes under the first field of view partition and the second field of view partition of the point cloud acquisition device, respectively. Acquire a first image. The first image is an image used to describe the first scene. The division module 202 is used to divide the first image into a plurality of sub-images. The plurality of sub-images include a first sub-image and a second sub-image. The first sub-image and the second sub-image are used to describe the first sub-scene and the second sub-scene, respectively. Each of the plurality of sub-images is divided into a plurality of image blocks. The fusion module 203 is used to fuse the first point cloud and the first image based on the plurality of image blocks. For example, combining Figure 4, the acquisition module 201 can be used to perform the receiving step in S103. The division module 202 can be used to perform S104-S105. The fusion module 203 can be used to perform S106. Figure 7 , the acquisition module 201 can be used to execute the receiving steps in S201, S202-S203. Fig. 9 The acquisition module 201 can be used to execute S300, and the adjustment module 204 can be used to execute S301 to S303.

[0184] Optionally, the division module 202 is specifically configured to: divide the first sub-image into a plurality of first image blocks, wherein different first image blocks have the same size; and divide the second sub-image into a plurality of second image blocks, wherein different second image blocks have the same size.

[0185] Optionally, the acquisition module 201 is specifically used to: acquire a target interval, which is an interval where the density of the registration points of the first sub-image is located. The registration point is a pixel point in the first image that has a corresponding scanning point in the first point cloud. The first pixel point in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinates of the first scanning point in the coordinate system of the first image are the same as the two-dimensional coordinates of the first pixel point in the coordinate system of the first image. According to the correspondence between each interval in a plurality of intervals and the image block size, the image block size corresponding to the target interval is acquired, and the acquired image block size is used as the first image block size. The division module 202 is specifically used to: divide the first sub-image into a plurality of first image blocks based on the first image block size.

[0186] Optionally, the acquisition module 201 is specifically used to: acquire a first graphic, where the first graphic is determined based on the registration points in the first sub-image. Determine the number of first image blocks based on the similarity between the first graphic and the target graphic. The division module 202 is specifically used to: divide the first sub-image into a plurality of first image blocks based on the number of first image blocks.

[0187] Optionally, the division module 202 is specifically configured to: extract a target object from the first sub-image, and divide the first sub-image into a plurality of first image blocks based on the extracted target object.

[0188] Optionally, the division module 202 is specifically configured to: treat each registration point in the first sub-image as a pixel point in the first image block, and assign the non-registration point to the first image block to which one of the registration points belongs based on the distance between the non-registration point in the first sub-image and each registration point.

[0189] Optionally, the acquisition module 201 is also used to: acquire multiple groups of first partition parameters based on the target point cloud. The target point cloud is a point cloud used to describe a second scene in an autonomous driving environment. The second scene is a scene before the first scene; different groups of first partition parameters are acquired based on different partitioning methods. Obtain the weight of each group of first partition parameters in the multiple groups of first partition parameters. Obtain the second partition parameters based on the multiple groups of first partition parameters and the weight of each group of first partition parameters. The division module 202 is also used to divide the field of view of the point cloud acquisition device into multiple field of view partitions using the second partition parameters. The multiple field of view partitions include a first field of view partition and a second field of view partition.

[0190] Optionally, the acquisition module 201 is further used to: acquire the resolution of the first image. The adjustment module 204 is used to: if the resolution of the first image is greater than the first threshold, then based on any one of the fused point cloud, the first point cloud or the first image, adjust at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition. The fused point cloud is a point cloud obtained by fusing the first image and the first point cloud. If the resolution of the first image is less than or equal to the first threshold, then based on the first point cloud or the fused point cloud, adjust at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition.

[0191] In one example, see Figure 3 The receiving function in the acquisition module 201 can be Figure 3 The communication interface 104 in the embodiment is implemented, and the above-mentioned division module 202, fusion module 203 and adjustment module 204 can all be implemented by Figure 3 The processor 101 in the embodiment calls the computer program stored in the memory 103 for implementation.

[0192] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation of any point cloud and image fusion device 20 provided above and the description of the beneficial effects can refer to the above corresponding method embodiments, which will not be repeated here.

[0193] It should be noted that the actions performed by the above modules are only specific examples. The actions actually performed by each unit refer to the above-mentioned Figure 4 , Figure 7 or Fig. 9 The actions or steps mentioned in the description of the embodiments are described.

[0194] An embodiment of the present application also provides an apparatus (such as a computer device or chip), including: a memory and a processor; the memory is used to store a computer program, and the processor is used to call the computer program to execute the actions or steps mentioned in any of the embodiments provided above.

[0195] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer is enabled to execute the actions or steps mentioned in any of the embodiments provided above.

[0196] The present application also provides a chip. The chip integrates a circuit and one or more interfaces for implementing the functions of the above-mentioned point cloud and image fusion device 20. Optionally, the functions supported by the chip may include: Figure 4 , Figure 7 or Fig. 9 The processing actions in the embodiments are not described in detail here. A person of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by instructing the relevant hardware through a program. The program can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a random access memory, etc. The processing unit or processor can be a central processing unit, a general-purpose processor, an application-specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0197] The embodiment of the present application also provides a computer program product including instructions, when the instructions are run on a computer, the computer executes any one of the methods in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that may be integrated with one or more available media. The available media may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0198] It should be noted that the above-mentioned devices for storing computer instructions or computer programs provided in the embodiments of the present application, such as but not limited to the above-mentioned memories, computer-readable storage media and communication chips, etc., are all non-transitory.

[0199] In the process of implementing the claimed application, those skilled in the art may understand and implement other changes to the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0200] Although the present application has been described in conjunction with specific features and embodiments thereof, various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application.

Claims

1. A method for fusing point cloud and image, characterized in that: Applied to a computer device, the method comprises: Acquire a first point cloud; wherein the first point cloud is a point cloud used to describe a first scene; the first point cloud includes a first sub-point cloud and a second sub-point cloud, the first sub-point cloud and the second sub-point cloud are used to describe a first sub-scene and a second sub-scene in the first scene, respectively, and the first sub-scene and the second sub-scene are sub-scenes under a first field of view partition and a second field of view partition of a point cloud acquisition device, respectively; Acquire a first image; wherein the first image is an image used to describe the first scene; Dividing the first image into a plurality of sub-images; wherein the plurality of sub-images include a first sub-image and a second sub-image, and the first sub-image and the second sub-image are used to describe the first sub-scene and the second sub-scene respectively; Dividing each sub-image of the plurality of sub-images into a plurality of image blocks; Based on the multiple image blocks, the first point cloud and the first image are fused.

2. The method according to claim 1, characterized in that: The step of dividing each of the plurality of sub-images into a plurality of image blocks comprises: Dividing the first sub-image into a plurality of first image blocks; wherein different first image blocks have the same size; The second sub-image is divided into a plurality of second image blocks; wherein different second image blocks have the same size.

3. The method according to claim 2, characterized in that The step of dividing the first sub-image into a plurality of first image blocks comprises: Acquire a target interval, where the target interval is an interval where the density of the registration points of the first sub-image is located; the registration points are pixels in the first image that have corresponding scanning points in the first point cloud; the first pixel in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinates of the first scanning point in the coordinate system of the first image are the same as the two-dimensional coordinates of the first pixel in the coordinate system of the first image; According to the correspondence between each interval in the plurality of intervals and the image block size, the image block size corresponding to the target interval is obtained, and the obtained image block size is used as the first image block size; Based on the first image block size, the first sub-image is divided into a plurality of first image blocks.

4. The method according to claim 2, characterized in that: The step of dividing the first sub-image into a plurality of first image blocks comprises: Acquire a first graphic, where the first graphic is determined based on the registration points in the first sub-image; Determining the number of the first image blocks based on the similarity between the first graphic and the target graphic; Based on the number of the first image blocks, the first sub-image is divided into a plurality of first image blocks.

5. The method according to claim 1, characterized in that The step of dividing each of the plurality of sub-images into a plurality of image blocks comprises: A target object is extracted from the first sub-image, and the first sub-image is divided into a plurality of first image blocks based on the extracted target object.

6. The method according to claim 1, characterized in that The step of dividing each of the plurality of sub-images into a plurality of image blocks comprises: Taking each registration point in the first sub-image as a pixel point in a first image block; Based on the distance between the non-registration point in the first sub-image and each of the registration points, the non-registration point is allocated to the first image block to which one of the registration points belongs.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire multiple groups of first partition parameters based on the target point cloud; wherein the target point cloud is a point cloud used to describe a second scene; the second scene is a scene before the first scene; different groups of first partition parameters are acquired based on different partitioning methods; Obtaining a weight of each group of first partition parameters in the multiple groups of first partition parameters; Based on the multiple groups of first partition parameters and the weight of each group of first partition parameters, obtaining second partition parameters; The field of view of the point cloud acquisition device is divided into a plurality of field of view partitions using the second partition parameter; wherein the plurality of field of view partitions include the first field of view partition and the second field of view partition.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire the resolution of the first image; If the resolution of the first image is greater than a first threshold, adjusting at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition based on any one of the fused point cloud, the first point cloud or the first image; wherein the fused point cloud is a point cloud obtained by fusing the first image and the first point cloud; If the resolution of the first image is less than or equal to the first threshold, at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition is adjusted based on the first point cloud or the fused point cloud.

9. A point cloud and image fusion device, characterized in that: The fusion device comprises: An acquisition module is used to acquire a first point cloud; wherein the first point cloud is a point cloud used to describe a first scene; the first point cloud includes a first sub-point cloud and a second sub-point cloud, the first sub-point cloud and the second sub-point cloud are used to describe a first sub-scene and a second sub-scene in the first scene, respectively, the first sub-scene and the second sub-scene are sub-scenes under a first field of view partition and a second field of view partition of a point cloud acquisition device, respectively; acquire a first image; wherein the first image is an image used to describe the first scene; a dividing module, configured to divide the first image into a plurality of sub-images; wherein the plurality of sub-images include a first sub-image and a second sub-image, the first sub-image and the second sub-image being used to describe the first sub-scene and the second sub-scene respectively; and divide each of the plurality of sub-images into a plurality of image blocks; A fusion module is used to fuse the first point cloud and the first image based on the multiple image blocks.

10. The fusion device according to claim 9, characterized in that: The division module is specifically used for: Dividing the first sub-image into a plurality of first image blocks; wherein different first image blocks have the same size; The second sub-image is divided into a plurality of second image blocks; wherein different second image blocks have the same size.

11. The fusion device according to claim 10, characterized in that: The acquisition module is specifically used for: Acquire a target interval, where the target interval is an interval where the density of the registration points of the first sub-image is located; the registration point is a pixel point in the first image that has a corresponding scanning point in the first point cloud; the first pixel point in the first image corresponds to the first scanning point in the first point cloud, which means that the two-dimensional coordinates of the first scanning point in the coordinate system of the first image are the same as the two-dimensional coordinates of the first pixel point in the coordinate system of the first image; acquire the image block size corresponding to the target interval according to the corresponding relationship between each interval in a plurality of intervals and the image block size, and use the acquired image block size as the first image block size; The division module is specifically configured to divide the first sub-image into a plurality of first image blocks based on the size of the first image block.

12. The fusion device according to claim 10, characterized in that: The acquisition module is specifically used for: Acquire a first graphic, where the first graphic is determined based on the registration points in the first sub-image; determine the number of the first image blocks based on the similarity between the first graphic and the target graphic; The division module is specifically configured to divide the first sub-image into a plurality of first image blocks based on the number of the first image blocks.

13. The fusion device according to claim 9, characterized in that: The partitioning module is specifically used for: A target object is extracted from the first sub-image, and the first sub-image is divided into a plurality of first image blocks based on the extracted target object.

14. The fusion device according to claim 9, characterized in that: The partitioning module is specifically used for: Taking each registration point in the first sub-image as a pixel point in a first image block; Based on the distance between the non-registration point in the first sub-image and each of the registration points, the non-registration point is allocated to the first image block to which one of the registration points belongs.

15. The fusion device according to any one of claims 9 to 14, characterized in that: The acquisition module is also used for: Acquire multiple groups of first partition parameters based on the target point cloud; wherein the target point cloud is a point cloud used to describe a second scene; the second scene is a scene before the first scene; different groups of first partition parameters are acquired based on different partitioning methods; Obtaining a weight of each group of first partition parameters in the multiple groups of first partition parameters; Based on the multiple groups of first partition parameters and the weight of each group of first partition parameters, obtaining second partition parameters; The division module is also used to divide the field of view of the point cloud acquisition device into multiple field of view partitions using the second partition parameter; wherein the multiple field of view partitions include the first field of view partition and the second field of view partition.

16. The fusion device according to any one of claims 9 to 14, characterized in that: The acquisition module is also used for: Acquire the resolution of the first image; The fusion device further comprises: an adjustment module; The adjustment module is used to: if the resolution of the first image is greater than a first threshold, adjust at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition based on any one of the fused point cloud, the first point cloud or the first image; wherein the fused point cloud is a point cloud obtained by fusing the first image and the first point cloud; If the resolution of the first image is less than or equal to the first threshold, at least one of the scanning density of the first field of view partition and the scanning density of the second field of view partition is adjusted based on the first point cloud or the fused point cloud.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 8.

18. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images