Three-dimensional video fusion method, device and electronic equipment
By selecting the target points with large HSV color difference in three-dimensional video fusion and determining the target color, the problem of low alignment accuracy between the three-dimensional model and two-dimensional image in the prior art is solved, and the color accuracy and alignment effect are improved.
Patent Information
- Application Number
- CN202410997034.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-23
AI Technical Summary
The existing three-dimensional video fusion method is sensitive to data noise when there are shadows, dark areas or low resolution of two-dimensional images, resulting in poor alignment of the three-dimensional model with the two-dimensional image and low color accuracy.
By acquiring the initial three-dimensional spatial model and two-dimensional image, select target three-dimensional points and pixel points with HSV color difference greater than the threshold, align and determine the target HSV color based on these points, generate the target three-dimensional spatial model to improve alignment accuracy and color accuracy.
Improves the color accuracy during the 3D video fusion process, reduces the deviation between the final 3D model color and the actual color, and enhances the alignment accuracy.
Smart Images

Figure CN118587399B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of video fusion technology, and in particular to a three-dimensional video fusion method, device and electronic equipment. Background Art
[0002] Current 3D video fusion methods typically stitch captured images into a panoramic image, then use a linear transformation algorithm to establish a mapping relationship between the 3D model and the 2D image, effectively aligning the 3D model and the 2D image. The 2D image's color is then assigned to each corresponding feature point in the 3D model to achieve 3D video fusion. Because linear transformation algorithms are highly sensitive to data noise, the presence of shadows, dark areas, or low resolution in the 2D image can directly affect the alignment between the 3D model and the 2D image, resulting in low color accuracy during the 3D video fusion process. Summary of the Invention
[0003] In view of this, the present application provides a 3D video fusion method, device, and electronic device to improve the color accuracy during the 3D video fusion process.
[0004] The technical solutions provided in this application are as follows:
[0005] According to an embodiment of the first aspect of the present application, a three-dimensional video fusion method is proposed, the method comprising:
[0006] Obtain an initial three-dimensional space model, the initial three-dimensional space model includes multiple three-dimensional points, each three-dimensional point corresponds to an initial HSV color;
[0007] Obtain a two-dimensional image, where the two-dimensional image includes multiple pixels, and each pixel corresponds to a reference HSV color;
[0008] Selecting multiple target 3D points from the multiple 3D points, wherein a difference between an initial HSV color corresponding to each target 3D point and an initial HSV color corresponding to an adjacent 3D point is greater than a first threshold;
[0009] Selecting multiple target pixels from the multiple pixels, wherein a difference between a reference HSV color corresponding to each target pixel and a reference HSV color corresponding to an adjacent pixel is greater than a second threshold;
[0010] Aligning the initial three-dimensional space model with the two-dimensional image based on the multiple target three-dimensional points and the multiple target pixel points to obtain a plurality of point pairs, each point pair including a three-dimensional point in the initial three-dimensional space model and a pixel point in the two-dimensional image;
[0011] For each point pair in the plurality of point pairs, determining a target HSV color corresponding to a three-dimensional point in the point pair based on a reference HSV color corresponding to a pixel point in the point pair;
[0012] A target three-dimensional space model is generated based on the target HSV colors corresponding to the three-dimensional points included in each point pair.
[0013] Optionally, obtaining the initial three-dimensional space model includes:
[0014] Collecting distance data of the current scene through a laser transmitter, wherein the distance data refers to the distance between the laser transmitter and a three-dimensional point corresponding to a reference object included in the current scene;
[0015] The pulse transmitter collects color reflection data of the current scene, wherein the color reflection data refers to color data obtained after the pulse emitted by the pulse transmitter is reflected by the three-dimensional point corresponding to the reference object; the laser transmitter and the pulse transmitter are configured with the same emission frequency and dot density;
[0016] The initial three-dimensional space model is determined according to the distance data and the color reflection data.
[0017] Optionally, selecting a plurality of target three-dimensional points from the plurality of three-dimensional points includes:
[0018] Determining a plurality of reference three-dimensional points from the plurality of three-dimensional points included in the initial three-dimensional space model, wherein a difference between an initial HSV color corresponding to each reference three-dimensional point and an initial HSV color corresponding to an adjacent three-dimensional point is greater than a first threshold;
[0019] According to the preset alignment accuracy, multiple target 3D points are selected from multiple reference 3D points;
[0020] The selecting a plurality of target pixel points from a plurality of pixel points of the two-dimensional image includes:
[0021] Determining a plurality of reference pixels from a plurality of pixels included in the two-dimensional image, wherein a difference between a reference HSV color corresponding to each reference pixel and a reference HSV color corresponding to an adjacent pixel is greater than a second threshold;
[0022] According to the preset alignment accuracy, multiple target pixels are selected from multiple reference pixels;
[0023] The alignment accuracy is used to indicate the number of the target three-dimensional points and the target pixel points.
[0024] Optionally, aligning the initial three-dimensional space model with the two-dimensional image based on the multiple target three-dimensional points and the multiple target pixel points includes:
[0025] For each target 3D point, determine the target pixel corresponding to the target 3D point based on the initial HSV color corresponding to the target 3D point, the initial HSV colors corresponding to the 3D points adjacent to the target 3D point, and the reference HSV colors corresponding to each target pixel;
[0026] The initial three-dimensional space model is aligned with the two-dimensional image according to the correspondence between each target three-dimensional point and the target pixel point.
[0027] Optionally, for each target 3D point, determining the target pixel corresponding to the target 3D point according to the initial HSV color corresponding to the target 3D point, the initial HSV colors corresponding to the 3D points adjacent to the target 3D point, and the reference HSV colors corresponding to each target pixel point includes:
[0028] For each target 3D point, if it is determined that the difference between the initial HSV color corresponding to the target 3D point and the reference HSV color corresponding to any target pixel is less than a third threshold, and the difference between the initial HSV color difference corresponding to the target 3D point and the reference HSV color difference corresponding to the target pixel is less than a fourth threshold, then the target pixel is determined to be the target pixel corresponding to the target 3D point;
[0029] Among them, the initial HSV color difference refers to the difference between the initial HSV color corresponding to the target 3D point and the initial HSV color corresponding to the adjacent 3D point, and the reference HSV color difference refers to the difference between the reference HSV color corresponding to the target pixel point and the reference HSV color corresponding to the adjacent pixel point.
[0030] Optionally, the point pairs include mutation point pairs and non-mutation point pairs, the mutation point pairs being point pairs consisting of the target three-dimensional point and the target pixel point; and for each point pair in the plurality of point pairs, determining the target HSV color corresponding to the three-dimensional point in the point pair based on the reference HSV color corresponding to the pixel point in the point pair includes:
[0031] For each mutation point pair, the reference HSV color corresponding to the pixel point in the mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the mutation point pair;
[0032] For each non-mutation point pair, the reference HSV color corresponding to the pixel point in the non-mutation point pair or the average value of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
[0033] Optionally, for each non-mutation point pair, taking the reference HSV color corresponding to the pixel point in the non-mutation point pair or the average value of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair, includes:
[0034] For each non-mutation point pair, determining the hue difference between the pixel point in the non-mutation point pair and the three-dimensional point;
[0035] If the hue difference is less than or equal to the preset hue difference threshold, the reference HSV color corresponding to the pixel point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair;
[0036] If the hue difference is greater than the preset hue difference threshold, the average of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
[0037] According to an embodiment of the second aspect of the present application, a three-dimensional video fusion device is proposed, which includes:
[0038] An acquisition unit is used to acquire an initial three-dimensional space model, where the initial three-dimensional space model includes a plurality of three-dimensional points, each of which corresponds to an initial HSV color;
[0039] Obtain a two-dimensional image, where the two-dimensional image includes multiple pixels, and each pixel corresponds to a reference HSV color;
[0040] a selection unit, configured to select a plurality of target three-dimensional points from the plurality of three-dimensional points, wherein a difference between an initial HSV color corresponding to each target three-dimensional point and an initial HSV color corresponding to an adjacent three-dimensional point is greater than a first threshold;
[0041] Selecting multiple target pixels from the multiple pixels, wherein a difference between a reference HSV color corresponding to each target pixel and a reference HSV color corresponding to an adjacent pixel is greater than a second threshold;
[0042] an alignment unit, configured to align the initial three-dimensional space model with the two-dimensional image based on the multiple target three-dimensional points and the multiple target pixel points, to obtain a plurality of point pairs, each point pair including a three-dimensional point in the initial three-dimensional space model and a pixel point in the two-dimensional image;
[0043] a determining unit configured to determine, for each point pair in the plurality of point pairs, a target HSV color corresponding to a three-dimensional point in the point pair based on a reference HSV color corresponding to a pixel point in the point pair;
[0044] A target three-dimensional space model is generated based on the target HSV colors corresponding to the three-dimensional points included in each point pair.
[0045] Optionally, the acquiring unit is specifically configured to:
[0046] Collecting distance data of the current scene through a laser transmitter, wherein the distance data refers to the distance between the laser transmitter and a three-dimensional point corresponding to a reference object included in the current scene;
[0047] The pulse transmitter collects color reflection data of the current scene, wherein the color reflection data refers to color data obtained after the pulse emitted by the pulse transmitter is reflected by the three-dimensional point corresponding to the reference object; the laser transmitter and the pulse transmitter are configured with the same emission frequency and dot density;
[0048] Determine the initial three-dimensional space model according to the distance data and the color reflection data;
[0049] And / or, the selection unit is specifically configured to:
[0050] Determining a plurality of reference three-dimensional points from the plurality of three-dimensional points included in the initial three-dimensional space model, wherein a difference between an initial HSV color corresponding to each reference three-dimensional point and an initial HSV color corresponding to an adjacent three-dimensional point is greater than a first threshold;
[0051] According to the preset alignment accuracy, multiple target 3D points are selected from multiple reference 3D points;
[0052] The selecting a plurality of target pixel points from a plurality of pixel points of the two-dimensional image includes:
[0053] Determining a plurality of reference pixels from a plurality of pixels included in the two-dimensional image, wherein a difference between a reference HSV color corresponding to each reference pixel and a reference HSV color corresponding to an adjacent pixel is greater than a second threshold;
[0054] According to the preset alignment accuracy, multiple target pixels are selected from multiple reference pixels;
[0055] The alignment accuracy is used to indicate the number of the target three-dimensional points and the target pixel points;
[0056] And / or, the alignment unit is specifically configured to:
[0057] For each target 3D point, determine the target pixel corresponding to the target 3D point based on the initial HSV color corresponding to the target 3D point, the initial HSV colors corresponding to the 3D points adjacent to the target 3D point, and the reference HSV colors corresponding to each target pixel;
[0058] aligning the initial three-dimensional space model with the two-dimensional image according to a correspondence between each target three-dimensional point and a target pixel point;
[0059] And / or, the alignment unit is specifically configured to:
[0060] For each target 3D point, if it is determined that the difference between the initial HSV color corresponding to the target 3D point and the reference HSV color corresponding to any target pixel is less than a third threshold, and the difference between the initial HSV color difference corresponding to the target 3D point and the reference HSV color difference corresponding to the target pixel is less than a fourth threshold, then the target pixel is determined to be the target pixel corresponding to the target 3D point;
[0061] The initial HSV color difference refers to the difference between the initial HSV color corresponding to the target 3D point and the initial HSV color corresponding to the adjacent 3D point; the reference HSV color difference refers to the difference between the reference HSV color corresponding to the target pixel and the reference HSV color corresponding to the adjacent pixel;
[0062] And / or, the point pair includes a mutation point pair and a non-mutation point pair, and the mutation point pair refers to a point pair consisting of the target three-dimensional point and the target pixel point; and the determining unit is specifically configured to:
[0063] For each mutation point pair, the reference HSV color corresponding to the pixel point in the mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the mutation point pair;
[0064] For each non-mutation point pair, the reference HSV color corresponding to the pixel point in the non-mutation point pair or the average of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the 3D point in the non-mutation point pair is used as the target HSV color corresponding to the 3D point in the non-mutation point pair;
[0065] And / or, the determining unit is specifically configured to:
[0066] For each non-mutation point pair, determining the hue difference between the pixel point in the non-mutation point pair and the three-dimensional point;
[0067] If the hue difference is less than or equal to the preset hue difference threshold, the reference HSV color corresponding to the pixel point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair;
[0068] If the hue difference is greater than the preset hue difference threshold, the average of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
[0069] According to an embodiment of the third aspect of the present application, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.
[0070] As can be seen from the above technical solutions, this application aligns the initial three-dimensional space model with the two-dimensional image based on multiple target three-dimensional points selected from the initial three-dimensional space model and multiple target pixel points selected from the two-dimensional image, and obtains multiple point pairs, wherein the target three-dimensional points and target pixel points are points where the HSV color mutates, and each point pair includes a three-dimensional point in the initial three-dimensional space model and a pixel point in the two-dimensional image. Further, for each point pair, the target HSV color corresponding to the three-dimensional point is determined based on the reference HSV color corresponding to the pixel point in the point pair, and finally the target three-dimensional space model is generated according to the target HSV colors corresponding to all three-dimensional points, thereby completing the fusion of the three-dimensional video. By aligning the initial three-dimensional space model and the two-dimensional image at the points where the HSV color mutates in the initial three-dimensional space model and the two-dimensional image, the alignment accuracy is improved, and the target HSV color corresponding to the three-dimensional point is determined by the reference HSV color corresponding to the pixel point in each point pair, so that the color of the target three-dimensional space model finally generated is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0072] Figure 1 A flowchart of a three-dimensional video fusion method provided in an embodiment of the present application;
[0073] Figure 2 A schematic diagram of the process of collecting distance data for the current scene provided in an embodiment of the present application;
[0074] Figure 3A Schematic diagram of the RGB color model provided in the embodiment of this application;
[0075] Figure 3B A schematic diagram of the HSV color model provided in an embodiment of the present application;
[0076] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0077] Figure 5 This is a structural diagram of a three-dimensional video fusion device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0078] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.
[0079] Three-dimensional video fusion technology is a technology that integrates video images (two-dimensional images) with three-dimensional models, realizing the combination of virtual scenes and real-time video images. It has been widely used in many fields such as security and smart home.
[0080] Current 3D video fusion technology, after acquiring a 2D image and building a 3D model, typically uses a linear transformation algorithm to establish a mapping relationship between the 3D model and the 2D image. Based on this mapping relationship, the 3D model and the 2D image are aligned. After the 3D model and 2D image are aligned, each pixel in the 2D image is mapped to a 3D point in the 3D model. The color attributes of each pixel in the 2D image, such as the RGB color, are assigned to the corresponding 3D point in the 3D model. Ultimately, a colored 3D model is obtained, completing the 3D video fusion.
[0081] However, the linear transformation algorithm is very sensitive to data noise. When there are shadows or dark areas in the two-dimensional image, or the resolution of the acquired two-dimensional image is low, the mapping relationship between the three-dimensional model established by the linear transformation algorithm and the two-dimensional image is less accurate, resulting in a decrease in the alignment accuracy between the two-dimensional image and the three-dimensional model, and ultimately causing the color of the obtained three-dimensional model to deviate significantly from the actual color.
[0082] Based on this, the present application proposes a three-dimensional video fusion method to improve the alignment accuracy of two-dimensional images and three-dimensional models, and reduce the deviation between the color of the final three-dimensional model and the actual color.
[0083] Please refer to Figure 1 , Figure 1 A flowchart of a three-dimensional video fusion method provided in an embodiment of the present application includes:
[0084] Step 101: Obtain an initial three-dimensional space model. The initial three-dimensional space model includes a plurality of three-dimensional points, and each three-dimensional point corresponds to an initial HSV color.
[0085] In this embodiment, the initial three-dimensional space model is a three-dimensional model with initial HSV colors. The initial three-dimensional space model is composed of a plurality of three-dimensional points, and each three-dimensional point corresponds to an initial HSV color.
[0086] In this embodiment, the method can be applied to the controller of an electronic device. The initial three-dimensional space model can be collected by a physical module deployed in the electronic device, or it can be collected by other devices and sent to the electronic device. This application does not limit this.
[0087] As an embodiment, the initial three-dimensional space model is acquired by a physical module deployed in the electronic device. The specific method for acquiring the initial three-dimensional space model may include:
[0088] Collecting distance data of the current scene through a laser transmitter, where the distance data refers to the distance between the laser transmitter and a three-dimensional point corresponding to a reference object included in the current scene;
[0089] The pulse transmitter collects color reflection data of the current scene. Color reflection data refers to the color data obtained after the pulse emitted by the pulse transmitter is reflected by the three-dimensional point corresponding to the reference object. The laser transmitter and the pulse transmitter are configured with the same emission frequency and dot density.
[0090] An initial three-dimensional space model is determined based on the distance data and the color reflection data.
[0091] In this embodiment, the deployed laser emitters and pulse emitters can be preconfigured to have the same emission frequency and dot density. The laser emitters collect distance data between the laser emitters and 3D points corresponding to reference objects (i.e., objects in the current scene), and the pulse emitters collect color data generated by reflections of pulses emitted by the pulse emitters from 3D points corresponding to the reference objects in the current scene. Furthermore, an initial 3D spatial model with initial HSV colors is determined based on the distance data and color data corresponding to each 3D point in the current scene.
[0092] For example, since RGB colors are generally used instead of HSV colors in electronic devices, the electronic device that executes the three-dimensional model fusion method proposed in this embodiment can be a camera device configured with a LiDAR dot matrix laser transceiver module and an RGB band pulse (such as a color LED) transceiver module. The LiDAR dot matrix laser transceiver module collects the distance data of the current scene, and the RGB band pulse transceiver module collects the color data of the current scene. The three-dimensional space model with RGB colors is finally determined. At this time, the RGB colors can be further converted into HSV colors.
[0093] Specifically, the process of collecting distance data of the current scene through the LiDAR dot matrix laser transceiver module may include:
[0094] The LiDAR laser transmitter emits laser pulses in the form of a dot-matrix laser beam. The laser pulses are reflected back after contacting objects in space. When the LiDAR laser receiver receives the reflected signal, it records the laser's flight time, that is, the time from emission to receipt of reflection, which is recorded as TOF.
[0095] The distance data between the object in the space and the LiDAR dot matrix laser transceiver module is calculated based on TOF. The calculation formula can be: distance d = TOF * 3 * 10^8.
[0096] Furthermore, a point cloud is established through the obtained set of distance data to obtain the distance data corresponding to each three-dimensional point in the current scene.
[0097] The following combination Figure 2 This article briefly introduces the process of collecting distance data of the current scene through the LiDAR dot matrix laser transceiver module.
[0098] Please refer to Figure 2 , Figure 2 Schematic diagram of the process of collecting distance data of the current scene provided in an embodiment of the present application.
[0099] like Figure 2 As shown, the control module (such as a controller in an electronic device) controls the LiDAR driver to drive the laser emitter to generate a laser pulse signal in the form of a dot matrix laser beam, and diffuses the generated laser pulse signal through a pulse diffuser to ensure that the laser pulse can cover a wider area.
[0100] The diffused dot matrix laser will be reflected after contacting the target object. The reflected dot matrix laser will be captured by the laser detector after passing through the lens. By further processing the data received by the laser detector, the distance data between the target object and the LiDAR dot matrix laser transceiver module is obtained.
[0101] Point cloud data is generated based on the distance data of the three-dimensional points corresponding to the target object. The point cloud data includes information such as the shape, position and distance of the target object.
[0102] Finally, the generated power data is transmitted back to the control module for subsequent processing.
[0103] This concludes Figure 2 Description of the acquisition process for the mid-range data.
[0104] In this embodiment, the process of collecting color data of the current scene through the RGB band pulse transceiver module may include:
[0105] The RGB pulse transmitter emits pulses containing the three primary color bands of red, green and blue. When the pulses come into contact with objects in the space, they reflect light of different intensities in each color band.
[0106] When an RGB pulse receiver (such as a photodetector) receives a reflected light signal, it determines the light intensity of the light signal and detects the light intensity of the reflected light signal in each color frequency band to determine the color data of the three-dimensional point corresponding to the object in space.
[0107] Furthermore, a point cloud is established through the obtained set of color data to obtain the color data corresponding to each three-dimensional point in the current scene.
[0108] Since the pulse emission frequency and dot density of the LiDAR laser emitter and the RGB pulse emitter are set to the same in advance, the initial three-dimensional space model with RGB colors can be determined based on the distance data and color data corresponding to each three-dimensional point in the current scene.
[0109] In this embodiment, after obtaining the initial three-dimensional space model with RGB colors, the RGB data corresponding to each three-dimensional point can be converted into initial HSV data.
[0110] Please refer to Figure 3A as well as Figure 3B , Figure 3A This is a schematic diagram of the RGB color model provided in the embodiment of the present application. Figure 3B A schematic diagram of the HSV color model provided in an embodiment of the present application.
[0111] like Figure 3A As shown in Figure 1, the RGB color model is a color representation method based on the three primary colors of red, green, and blue. By adjusting the proportions of these three basic colors, various colors can be mixed.
[0112] like Figure 3B As shown, the HSV color model is based on the color wheel. Hue represents the position of the color in the rainbow, saturation represents the purity or grayscale of the color, and value represents the brightness of the color. It is a more intuitive way to describe color.
[0113] In this embodiment, the conversion formula for converting the RGB color model to the HSV color model may include:
[0114] R'=R / 255
[0115] G'=G / 255
[0116] B'=B / 255
[0117] C max =max(R', G', B')
[0118] C min =min(R',G',B')
[0119] Δ=C max -C min
[0120] tone:
[0121]
[0122] Saturation:
[0123]
[0124] Brightness:
[0125] V=C max
[0126] At this point, the description of step 101 ends, and step 102 is executed next.
[0127] Step 102: Acquire a two-dimensional image. The two-dimensional image includes a plurality of pixels, and each pixel corresponds to a reference HSV color.
[0128] In this embodiment, the two-dimensional image can be collected by a physical module deployed in the electronic device, or it can be collected by other devices and sent to the electronic device. This application does not limit this.
[0129] In this embodiment, the acquired two-dimensional image is a color image, and the two-dimensional image is composed of a plurality of pixels, and each pixel corresponds to a reference HSV color.
[0130] It should be noted that since electronic devices usually record colors in RGB color mode, if each pixel of the acquired two-dimensional image corresponds to an RGB color, the RGB color can be converted into a reference HSV color according to the formula for converting the RGB color into the HSV color in step 101.
[0131] At this point, the description of step 102 ends, and step 103 is executed next.
[0132] Step 103 : selecting a plurality of target 3D points from the plurality of 3D points, wherein a difference between an initial HSV color corresponding to each target 3D point and an initial HSV color corresponding to an adjacent 3D point is greater than a first threshold.
[0133] In this embodiment, multiple target three-dimensional points can be selected from the three-dimensional points included in the initial three-dimensional space model obtained in step 101. The target three-dimensional point refers to a three-dimensional point whose initial HSV color corresponding to the point and the initial HSV color corresponding to the adjacent three-dimensional point have a difference greater than a first threshold. In other words, the target three-dimensional point refers to a three-dimensional point in the initial three-dimensional space model where the initial HSV color undergoes a mutation.
[0134] The selection of the target three-dimensional point will be described in detail below and will not be repeated here.
[0135] At this point, the description of step 103 ends, and step 104 is executed next.
[0136] Step 104 : selecting a plurality of target pixels from the plurality of pixels, wherein a difference between a reference HSV color corresponding to each target pixel and a reference HSV color corresponding to an adjacent pixel is greater than a second threshold.
[0137] In this embodiment, multiple target pixel points can be selected from the pixel points included in the two-dimensional image obtained in step 102. The target pixel point refers to a pixel point whose reference HSV color corresponding to the pixel point and the reference HSV color corresponding to the adjacent pixel point have a difference greater than a second threshold. In other words, the target pixel point refers to a pixel point in the two-dimensional image where the reference HSV color undergoes a sudden change.
[0138] As an embodiment, a specific method of selecting multiple target 3D points from multiple 3D points may include:
[0139] Determining a plurality of reference three-dimensional points from the plurality of three-dimensional points included in the initial three-dimensional space model, wherein a difference between an initial HSV color corresponding to each reference three-dimensional point and an initial HSV color corresponding to an adjacent three-dimensional point is greater than a first threshold;
[0140] According to the preset alignment accuracy, multiple target 3D points are selected from multiple reference 3D points;
[0141] Exemplarily, a specific method of selecting multiple target pixel points from multiple pixel points of a two-dimensional image may include:
[0142] Determining a plurality of reference pixels from a plurality of pixels included in the two-dimensional image, wherein a difference between a reference HSV color corresponding to each reference pixel and a reference HSV color corresponding to an adjacent pixel is greater than a second threshold;
[0143] According to the preset alignment accuracy, multiple target pixels are selected from multiple reference pixels;
[0144] The alignment accuracy is used to indicate the number of target 3D points and target pixels.
[0145] In this embodiment, after presetting the first threshold and the second threshold, multiple reference three-dimensional points that meet the initial HSV value mutation can be determined from the multiple three-dimensional points included in the initial three-dimensional space model, and multiple reference pixel points that meet the reference HSV value mutation can be determined from the multiple pixel points included in the two-dimensional image.
[0146] It is easy to understand that since the three-dimensional space model and the two-dimensional image need to be aligned subsequently based on the three-dimensional points with sudden changes in HSV values in the initial three-dimensional space model and the pixel points with sudden changes in HSV values in the two-dimensional image, the more three-dimensional points and pixel points involved in the alignment process, the higher the alignment accuracy and the better the alignment effect. However, the participation of too many three-dimensional points and pixel points in the alignment process places extremely high requirements on the computing power and resources of the controller of the electronic device. Therefore, the alignment accuracy can be pre-set according to the computing power of the electronic device controller and the accuracy requirements in actual applications. The alignment accuracy is used to indicate the number of target three-dimensional points and target pixel points participating in the alignment process of the three-dimensional model and the two-dimensional image.
[0147] In this embodiment, the target 3D points and target pixels can be selected by determining them from a plurality of reference 3D points and a plurality of reference pixels at equal intervals. The higher the alignment accuracy, the more target 3D points and target pixels are selected, and the smaller the intervals between the selected target 3D points and target pixels are.
[0148] For example, assuming the number of determined reference 3D points and reference pixels is 100, and the preset alignment accuracy is 50, that is, 50 reference 3D points are selected as target 3D points, and 50 reference pixels are selected as target pixels. In this case, the selection method can be to select a target 3D point from every other reference 3D point of the 100 reference 3D points, resulting in 50 target 3D points; and to select a target pixel from every other reference pixel of the 100 reference pixels, resulting in 50 target pixels. It is easy to understand that the higher the alignment accuracy, the smaller the interval between the selected points; the lower the alignment accuracy, the larger the interval between the selected points.
[0149] In this embodiment, after determining the reference three-dimensional points that meet the initial HSV value mutation and the reference pixel points that meet the reference HSV value mutation, the number of target three-dimensional points and target pixel points that need to participate in the alignment process can be further determined according to the preset alignment accuracy, and the corresponding number of target three-dimensional points can be obtained from the reference three-dimensional points that meet the initial HSV value mutation, and the corresponding number of target pixel points can be obtained from the reference pixel points that meet the reference HSV value mutation.
[0150] At this point, the description of step 104 ends, and step 105 is executed next.
[0151] Step 105 , aligning the initial 3D space model with the 2D image based on multiple target 3D points and multiple target pixel points to obtain multiple point pairs, each point pair including a 3D point in the initial 3D space model and a pixel point in the 2D image.
[0152] In this embodiment, the initial three-dimensional space model and the two-dimensional image are aligned according to the multiple target three-dimensional points and multiple target pixel points obtained in step 103 and step 104. After alignment, multiple point pairs are obtained, and each point pair includes a three-dimensional point in the initial three-dimensional space model and a pixel point in the two-dimensional image, indicating that the three-dimensional point in the initial three-dimensional space model and the pixel point in the two-dimensional image actually correspond to the same point in the current scene.
[0153] As an embodiment, a method for aligning an initial three-dimensional space model with a two-dimensional image based on multiple target three-dimensional points and multiple target pixel points may include:
[0154] For each target 3D point, determine the target pixel corresponding to the target 3D point based on the initial HSV color corresponding to the target 3D point, the initial HSV colors corresponding to the 3D points adjacent to the target 3D point, and the reference HSV colors corresponding to each target pixel;
[0155] According to the correspondence between each target 3D point and the target pixel point, the initial 3D space model is aligned with the 2D image.
[0156] In this embodiment, the target pixel point corresponding to the target 3D point can be determined based on the initial HSV color corresponding to the target 3D point, the initial HSV colors corresponding to the 3D points adjacent to the target 3D point, and the reference HSV colors corresponding to each target pixel point.
[0157] After determining the correspondence between each target 3D point and the target pixel point, the initial 3D space model and the 2D image may be aligned according to the correspondence between the multiple target 3D points and the target pixel points.
[0158] As an embodiment, for each target 3D point, a specific method for determining a target pixel corresponding to the target 3D point based on the initial HSV color corresponding to the target 3D point, the initial HSV colors corresponding to the 3D points adjacent to the target 3D point, and the reference HSV colors corresponding to each target pixel may include:
[0159] For each target 3D point, if it is determined that the difference between the initial HSV color corresponding to the target 3D point and the reference HSV color corresponding to any target pixel is less than a third threshold, and the difference between the initial HSV color difference corresponding to the target 3D point and the reference HSV color difference corresponding to the target pixel is less than a fourth threshold, then the target pixel is determined to be the target pixel corresponding to the target 3D point;
[0160] Among them, the initial HSV color difference refers to the difference between the initial HSV color corresponding to the target 3D point and the initial HSV color corresponding to the adjacent 3D point, and the reference HSV color difference refers to the difference between the reference HSV color corresponding to the target pixel point and the reference HSV color corresponding to the adjacent pixel point.
[0161] In this embodiment, for each target three-dimensional point, the initial HSV color corresponding to the target three-dimensional point can be determined first, and target pixel points whose difference between the reference HSV color and the initial HSV color corresponding to the target three-dimensional point is less than a third threshold are selected from all target pixel points. The reference HSV colors corresponding to these target pixel points are similar to the initial HSV colors corresponding to the target three-dimensional point, indicating that these target pixel points may be the target pixel points corresponding to the target three-dimensional point.
[0162] Furthermore, the difference between the initial HSV color corresponding to the target 3D point and the initial HSV color corresponding to the adjacent 3D point is detected, and the difference is recorded as the initial HSV color difference. At the same time, the difference between the reference HSV color corresponding to each target pixel point close to the initial HSV color corresponding to the target 3D point and the reference HSV color corresponding to the adjacent pixel point is detected, and the difference is recorded as the reference HSV color difference.
[0163] It is easy to understand that if the difference between the initial HSV color difference and the reference HSV color difference is less than the fourth threshold, that is, the difference between the initial HSV color difference and the reference HSV color difference is very small, then it indicates that not only the HSV color of the target pixel point is similar to that of the target three-dimensional point, but the HSV color difference between it and the adjacent pixel points is also similar to the color difference between the target three-dimensional point and the adjacent three-dimensional points, indicating that the target pixel point is the target pixel point corresponding to the target three-dimensional point.
[0164] After determining the correspondence between each target three-dimensional point and the target pixel point according to the above method, the initial three-dimensional space model and the two-dimensional image are aligned according to the correspondence between each target three-dimensional point and the target pixel point, and the correspondence between each three-dimensional point in the initial three-dimensional space model and the pixel point in the two-dimensional image is obtained, that is, multiple point pairs are obtained, each point pair includes a three-dimensional point in the three-dimensional space model and a pixel point in the two-dimensional image.
[0165] At this point, the description of step 105 ends, and step 106 is executed next.
[0166] Step 106 : For each point pair in the plurality of point pairs, determine a target HSV color corresponding to the three-dimensional point in the point pair based on a reference HSV color corresponding to the pixel point in the point pair.
[0167] In this embodiment, for each point pair obtained in step 105 , the target HSV color corresponding to the three-dimensional point in the point pair can be determined based on the reference HSV color corresponding to the pixel point in the point pair.
[0168] As an embodiment, the point pairs include mutation point pairs and non-mutation point pairs. The mutation point pairs refer to point pairs consisting of a target three-dimensional point and a target pixel point. For each point pair, a method for determining a target HSV color corresponding to the three-dimensional point in the point pair based on a reference HSV color corresponding to the pixel point in the point pair may include:
[0169] For each mutation point pair, the reference HSV color corresponding to the pixel point in the mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the mutation point pair;
[0170] For each non-mutation point pair, the reference HSV color corresponding to the pixel point in the non-mutation point pair or the average value of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
[0171] In this embodiment, the multiple point pairs determined are divided into mutation point pairs and non-mutation point pairs, where the mutation point pairs refer to point pairs consisting of target three-dimensional points and target pixel points, and the non-mutation point pairs refer to point pairs consisting of non-target three-dimensional points and non-target pixel points.
[0172] For different types of point pairs, the method of determining the target HSV color corresponding to the three-dimensional point in the point pair is also different.
[0173] Specifically, since the mutation point pairs are obtained by screening in step 105 during alignment, the alignment accuracy is relatively high. Therefore, for the mutation point pairs, the reference HSV color corresponding to the target pixel point in the mutation point pair can be directly used as the target HSV color corresponding to the target three-dimensional point in the point pair.
[0174] For a non-mutation point pair, the reference HSV color corresponding to the pixel point in the non-mutation point pair or the average value of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair can be used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
[0175] As an embodiment, for each non-mutation point pair, a hue difference between a pixel point in the non-mutation point pair and a three-dimensional point is determined;
[0176] If the hue difference is less than or equal to the preset hue difference threshold, the reference HSV color corresponding to the pixel point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair;
[0177] If the hue difference is greater than the preset hue difference threshold, the average of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
[0178] In this embodiment, for each non-mutation point pair, the hue difference between the pixel point and the three-dimensional point in the non-mutation point pair can be determined first. If the hue difference is less than or equal to the preset hue difference threshold, it indicates that the color difference between the pixel point and the three-dimensional point in the non-mutation point pair is not much. At this time, the reference HSV color corresponding to the pixel point can be directly used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
[0179] If the determined hue difference value is greater than the preset hue difference threshold, it indicates that the color difference between the pixel point and the three-dimensional point in the non-mutation point pair is large. At this time, the average value of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair can be used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair to perform HSV color balancing.
[0180] At this point, the description of step 106 ends, and step 107 is executed next.
[0181] Step 107 : Generate a target three-dimensional space model based on the target HSV colors corresponding to the three-dimensional points included in each point pair.
[0182] In this embodiment, after determining the target HSV color corresponding to each 3D point, a target 3D space model is generated based on the target HSV color corresponding to each 3D point, thus completing the fusion of the 3D video and obtaining a target 3D space model with more accurate colors.
[0183] It should be noted that since the color model commonly used in electronic devices is the RGB color model, the target HSV color can be converted into RGB color and then the target three-dimensional space model is generated to obtain a three-dimensional space model with RGB color that is more suitable for processing by electronic devices.
[0184] As an example, a conversion formula for converting an HSV color to an RGB color may include:
[0185] C=V×S
[0186] X = C × (1 - | (H / 60°) mod 2 - 1 |)
[0187] m=VC
[0188]
[0189] (R,G,B)=((R'+m)×255,(G'+m)×255,(B'+m)×255)
[0190] This concludes the description of step 107.
[0191] This concludes Figure 1 Description of the process of the three-dimensional video fusion method.
[0192] This application aligns the initial three-dimensional space model with the two-dimensional image based on multiple target three-dimensional points selected from the initial three-dimensional space model and multiple target pixel points selected from the two-dimensional image, and obtains multiple point pairs, wherein the target three-dimensional points and target pixel points are points where the HSV color mutates, and each point pair includes a three-dimensional point in the initial three-dimensional space model and a pixel point in the two-dimensional image. Further, for each point pair, the target HSV color corresponding to the three-dimensional point is determined based on the reference HSV color corresponding to the pixel point in the point pair, and finally the target three-dimensional space model is generated according to the target HSV colors corresponding to all three-dimensional points, completing the fusion of the three-dimensional video. By aligning the initial three-dimensional space model and the two-dimensional image at the points where the HSV color mutates in the initial three-dimensional space model and the two-dimensional image, the alignment accuracy is improved, and the target HSV color corresponding to the three-dimensional point is determined by the reference HSV color corresponding to the pixel point in each point pair, so that the color of the target three-dimensional space model finally generated is more accurate.
[0193] Furthermore, current methods for implementing 3D video fusion require real-time image acquisition, splicing, point alignment, and color overlay, all performed at 24 Hz (the frequency at which the human eye produces a continuous visual effect). This application divides video fusion into two steps. Because a 3D spatial model with color is acquired, when objects in the space change, only the HSV values that have changed can be updated, reducing the update frequency. Only the color value overlay needs to maintain a 24 Hz frequency, thereby reducing the computational requirements of the controller.
[0194] At the same time, the color value acquisition process in the scheme proposed in this application is basically a comparison and averaging operation, which does not require complex algorithm calculations and has lower computational complexity. In addition, this scheme distributes the work of collecting color attributes to the LiDAR module by preliminarily collecting color attributes and subsequently covering the color of the video screen, further reducing the computational performance requirements for each module.
[0195] Please refer to Figure 4 , Figure 4It is a schematic structural diagram of an electronic device proposed in an embodiment of the present application. At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a terminal interaction device at the logical level. Of course, in addition to software implementation methods, this application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0196] Please refer to Figure 5 , Figure 5 This is a structural diagram of a three-dimensional video fusion device proposed in the embodiment of this application. Figure 5 As shown, the 3D video fusion device may include an acquisition unit 501, a selection unit 502, an alignment unit 503, and a determination unit 504. Specifically, the device includes:
[0197] An acquisition unit 501 is configured to acquire an initial three-dimensional space model, where the initial three-dimensional space model includes a plurality of three-dimensional points, each of which corresponds to an initial HSV color.
[0198] Obtain a two-dimensional image, where the two-dimensional image includes multiple pixels, and each pixel corresponds to a reference HSV color;
[0199] A selection unit 502 is configured to select a plurality of target 3D points from the plurality of 3D points, wherein a difference between an initial HSV color corresponding to each target 3D point and an initial HSV color corresponding to an adjacent 3D point is greater than a first threshold;
[0200] Selecting multiple target pixels from the multiple pixels, where the difference between the reference HSV color corresponding to each target pixel and the reference HSV color corresponding to an adjacent pixel is greater than a second threshold;
[0201] an alignment unit 503 for aligning the initial 3D spatial model with the 2D image based on the multiple target 3D points and the multiple target pixel points, to obtain multiple point pairs, each point pair including a 3D point in the initial 3D spatial model and a pixel point in the 2D image;
[0202] A determining unit 504 is configured to determine, for each point pair in the plurality of point pairs, a target HSV color corresponding to a three-dimensional point in the point pair based on a reference HSV color corresponding to a pixel point in the point pair;
[0203] Generate a target 3D space model based on the target HSV colors corresponding to all 3D points.
[0204] Optionally, the acquiring unit 501 is specifically configured to:
[0205] Collecting distance data of the current scene through a laser transmitter, where the distance data refers to the distance between the laser transmitter and a three-dimensional point corresponding to a reference object included in the current scene;
[0206] The pulse transmitter collects color reflection data of the current scene. Color reflection data refers to the color data obtained after the pulse emitted by the pulse transmitter is reflected by the three-dimensional point corresponding to the reference object. The laser transmitter and the pulse transmitter are configured with the same emission frequency and dot density.
[0207] Determine an initial three-dimensional space model based on the distance data and the color reflection data;
[0208] And / or, the selection unit 502 is specifically configured to:
[0209] Determining a plurality of reference three-dimensional points from the plurality of three-dimensional points included in the initial three-dimensional space model, wherein a difference between an initial HSV color corresponding to each reference three-dimensional point and an initial HSV color corresponding to an adjacent three-dimensional point is greater than a first threshold;
[0210] According to the preset alignment accuracy, multiple target 3D points are selected from multiple reference 3D points;
[0211] Select multiple target pixels from multiple pixels of a two-dimensional image, including:
[0212] Determining a plurality of reference pixels from a plurality of pixels included in the two-dimensional image, wherein a difference between a reference HSV color corresponding to each reference pixel and a reference HSV color corresponding to an adjacent pixel is greater than a second threshold;
[0213] According to the preset alignment accuracy, multiple target pixels are selected from multiple reference pixels;
[0214] Among them, the alignment accuracy is used to indicate the number of target 3D points and target pixels;
[0215] And / or, the alignment unit 503 is specifically configured to:
[0216] For each target 3D point, determine the target pixel corresponding to the target 3D point based on the initial HSV color corresponding to the target 3D point, the initial HSV colors corresponding to the 3D points adjacent to the target 3D point, and the reference HSV colors corresponding to each target pixel;
[0217] Align the initial 3D spatial model with the 2D image based on the correspondence between each target 3D point and the target pixel point;
[0218] And / or, the alignment unit 503 is specifically configured to:
[0219] For each target 3D point, if it is determined that the difference between the initial HSV color corresponding to the target 3D point and the reference HSV color corresponding to any target pixel is less than a third threshold, and the difference between the initial HSV color difference corresponding to the target 3D point and the reference HSV color difference corresponding to the target pixel is less than a fourth threshold, then the target pixel is determined to be the target pixel corresponding to the target 3D point;
[0220] The initial HSV color difference refers to the difference between the initial HSV color corresponding to the target 3D point and the initial HSV color corresponding to the adjacent 3D point. The reference HSV color difference refers to the difference between the reference HSV color corresponding to the target pixel and the reference HSV color corresponding to the adjacent pixel.
[0221] And / or, the point pair includes a mutation point pair and a non-mutation point pair, and the mutation point pair refers to a point pair consisting of a target three-dimensional point and a target pixel point; the determining unit 504 is specifically configured to:
[0222] For each mutation point pair, the reference HSV color corresponding to the pixel point in the mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the mutation point pair;
[0223] For each non-mutation point pair, the reference HSV color corresponding to the pixel point in the non-mutation point pair or the average of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the 3D point in the non-mutation point pair is used as the target HSV color corresponding to the 3D point in the non-mutation point pair;
[0224] And / or, the determining unit 504 is specifically configured to:
[0225] For each non-mutation point pair, determining the hue difference between the pixel point in the non-mutation point pair and the three-dimensional point;
[0226] If the hue difference is less than or equal to the preset hue difference threshold, the reference HSV color corresponding to the pixel point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair;
[0227] If the hue difference is greater than the preset hue difference threshold, the average of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
[0228] So far, completed Figure 5 Description of the three-dimensional video fusion device.
[0229] Correspondingly, in this embodiment, the embodiment of the present application further provides a computer-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed, the method disclosed in the above example of the present application can be implemented.
[0230] Exemplarily, the computer-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the computer-readable storage medium may be RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.
[0231] The above are merely preferred embodiments of the present application and are not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.
Claims
1. A three-dimensional video fusion method, It is characterized in that The method includes: Obtain an initial three-dimensional space model, the initial three-dimensional space model includes multiple three-dimensional points, each three-dimensional point corresponds to an initial HSV color; Obtain a two-dimensional image, the two-dimensional image includes multiple pixels, and each pixel corresponds to a reference HSV color; Selecting a plurality of target 3D points from the plurality of 3D points, wherein a difference between an initial HSV color corresponding to each target 3D point and an initial HSV color corresponding to an adjacent 3D point is greater than a first threshold; Selecting multiple target pixels from the multiple pixels, wherein the difference between the reference HSV color corresponding to each target pixel and the reference HSV color corresponding to an adjacent pixel is greater than a second threshold; For each target 3D point, if it is determined that the difference between the initial HSV color corresponding to the target 3D point and the reference HSV color corresponding to any target pixel point is less than the third threshold, and the difference between the initial HSV color difference corresponding to the target 3D point and the reference HSV color difference corresponding to the target pixel point is less than the fourth threshold, then the target pixel point is determined to be the target pixel point corresponding to the target 3D point; the initial HSV color difference refers to the difference between the initial HSV color corresponding to the target 3D point and the initial HSV color corresponding to the adjacent 3D point, and the reference HSV color difference refers to the difference between the reference HSV color corresponding to the target pixel point and the reference HSV color corresponding to the adjacent pixel point; According to the correspondence between each target three-dimensional point and a target pixel point, the initial three-dimensional space model is aligned with the two-dimensional image to obtain a plurality of point pairs, each point pair including a three-dimensional point in the initial three-dimensional space model and a pixel point in the two-dimensional image; For each point pair in the plurality of point pairs, determining a target HSV color corresponding to a three-dimensional point in the point pair based on a reference HSV color corresponding to a pixel point in the point pair; A target three-dimensional space model is generated based on the target HSV colors corresponding to the three-dimensional points included in each point pair.
2. The method according to claim 1, It is characterized in that The obtaining of the initial three-dimensional space model comprises: Collecting distance data of the current scene through a laser transmitter, wherein the distance data refers to the distance between the laser transmitter and a three-dimensional point corresponding to a reference object included in the current scene; The color reflection data of the current scene is collected by a pulse transmitter, wherein the color reflection data refers to the color data obtained after the pulse emitted by the pulse transmitter is reflected by the three-dimensional point corresponding to the reference object; the laser transmitter and the pulse transmitter are configured with the same emission frequency and dot matrix density; The initial three-dimensional space model is determined according to the distance data and the color reflection data.
3. The method according to claim 1, It is characterized in that The selecting a plurality of target three-dimensional points from the plurality of three-dimensional points comprises: Determine a plurality of reference three-dimensional points from the plurality of three-dimensional points included in the initial three-dimensional space model, wherein a difference between an initial HSV color corresponding to each reference three-dimensional point and an initial HSV color corresponding to an adjacent three-dimensional point is greater than a first threshold; According to a preset alignment accuracy, multiple target 3D points are selected from multiple reference 3D points; The selecting a plurality of target pixel points from a plurality of pixel points of the two-dimensional image comprises: Determine a plurality of reference pixels from a plurality of pixels included in the two-dimensional image, wherein a difference between a reference HSV color corresponding to each reference pixel and a reference HSV color corresponding to an adjacent pixel is greater than a second threshold; According to a preset alignment accuracy, multiple target pixel points are selected from multiple reference pixel points; The alignment accuracy is used to indicate the number of the target three-dimensional points and the target pixel points.
4. The method according to claim 1, It is characterized in that The point pairs include mutation point pairs and non-mutation point pairs, and the mutation point pairs refer to point pairs consisting of the target three-dimensional point and the target pixel point; for each point pair in the plurality of point pairs, determining the target HSV color corresponding to the three-dimensional point in the point pair based on the reference HSV color corresponding to the pixel point in the point pair includes: For each mutation point pair, the reference HSV color corresponding to the pixel point in the mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the mutation point pair; For each non-mutation point pair, the reference HSV color corresponding to the pixel point in the non-mutation point pair or the average value of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
5. The method according to claim 4, It is characterized in that The method of taking, for each non-mutation point pair, a reference HSV color corresponding to a pixel point in the non-mutation point pair or an average value of a reference HSV color corresponding to a pixel point in the non-mutation point pair and an initial HSV color of a three-dimensional point in the non-mutation point pair as a target HSV color corresponding to the three-dimensional point in the non-mutation point pair, comprises: For each non-mutation point pair, determine the hue difference between the pixel point in the non-mutation point pair and the three-dimensional point; If the hue difference value is less than or equal to the preset hue difference threshold, the reference HSV color corresponding to the pixel point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair; If the hue difference value is greater than the preset hue difference threshold, the average value of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
6. A three-dimensional video fusion device, It is characterized in that The device includes: An acquisition unit, used for acquiring an initial three-dimensional space model, the initial three-dimensional space model includes a plurality of three-dimensional points, each of which corresponds to an initial HSV color; Obtain a two-dimensional image, the two-dimensional image includes multiple pixels, and each pixel corresponds to a reference HSV color; A selection unit, configured to select a plurality of target three-dimensional points from the plurality of three-dimensional points, wherein a difference between an initial HSV color corresponding to each target three-dimensional point and an initial HSV color corresponding to an adjacent three-dimensional point is greater than a first threshold; Selecting multiple target pixel points from the multiple pixel points, the difference between the reference HSV color corresponding to each target pixel point and the reference HSV color corresponding to the adjacent pixel point is greater than a second threshold; An alignment unit, for each target 3D point, if it is determined that the difference between the initial HSV color corresponding to the target 3D point and the reference HSV color corresponding to any target pixel point is less than a third threshold, and the difference between the initial HSV color difference corresponding to the target 3D point and the reference HSV color difference corresponding to the target pixel point is less than a fourth threshold, then determining that the target pixel point is the target pixel point corresponding to the target 3D point; the initial HSV color difference refers to the difference between the initial HSV color corresponding to the target 3D point and the initial HSV color corresponding to the adjacent 3D point, and the reference HSV color difference refers to the difference between the reference HSV color corresponding to the target pixel point and the reference HSV color corresponding to the adjacent pixel point; According to the correspondence between each target three-dimensional point and a target pixel point, the initial three-dimensional space model is aligned with the two-dimensional image to obtain a plurality of point pairs, each point pair including a three-dimensional point in the initial three-dimensional space model and a pixel point in the two-dimensional image; a determining unit, configured to determine, for each point pair in the plurality of point pairs, a target HSV color corresponding to a three-dimensional point in the point pair based on a reference HSV color corresponding to a pixel point in the point pair; A target three-dimensional space model is generated based on the target HSV colors corresponding to the three-dimensional points included in each point pair.
7. The device according to claim 6, It is characterized in that The acquisition unit is specifically used for: Collecting distance data of the current scene through a laser transmitter, wherein the distance data refers to the distance between the laser transmitter and a three-dimensional point corresponding to a reference object included in the current scene; The color reflection data of the current scene is collected by a pulse transmitter, wherein the color reflection data refers to the color data obtained after the pulse emitted by the pulse transmitter is reflected by the three-dimensional point corresponding to the reference object; the laser transmitter and the pulse transmitter are configured with the same emission frequency and dot matrix density; Determine the initial three-dimensional space model according to the distance data and the color reflection data; And / or, the selection unit is specifically used for: Determine a plurality of reference three-dimensional points from the plurality of three-dimensional points included in the initial three-dimensional space model, wherein a difference between an initial HSV color corresponding to each reference three-dimensional point and an initial HSV color corresponding to an adjacent three-dimensional point is greater than a first threshold; According to a preset alignment accuracy, multiple target 3D points are selected from multiple reference 3D points; The selecting a plurality of target pixel points from a plurality of pixel points of the two-dimensional image comprises: Determine a plurality of reference pixels from a plurality of pixels included in the two-dimensional image, wherein a difference between a reference HSV color corresponding to each reference pixel and a reference HSV color corresponding to an adjacent pixel is greater than a second threshold; According to a preset alignment accuracy, multiple target pixel points are selected from multiple reference pixel points; Wherein, the alignment accuracy is used to indicate the number of the target three-dimensional points and the target pixel points; And / or, the point pair includes a mutation point pair and a non-mutation point pair, and the mutation point pair refers to a point pair consisting of the target three-dimensional point and the target pixel point; and the determining unit is specifically used for: For each mutation point pair, the reference HSV color corresponding to the pixel point in the mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the mutation point pair; For each non-mutation point pair, the reference HSV color corresponding to the pixel point in the non-mutation point pair or the average value of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair; And / or, the determining unit is specifically configured to: For each non-mutation point pair, determine the hue difference between the pixel point in the non-mutation point pair and the three-dimensional point; If the hue difference value is less than or equal to the preset hue difference threshold, the reference HSV color corresponding to the pixel point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair; If the hue difference value is greater than the preset hue difference threshold, the average value of the reference HSV color corresponding to the pixel point in the non-mutation point pair and the initial HSV color of the three-dimensional point in the non-mutation point pair is used as the target HSV color corresponding to the three-dimensional point in the non-mutation point pair.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.
