Thermal three-dimensional mapping method and device, server and readable storage medium

Abstract

The invention is suitable for the technical field of three-dimensional mapping, and provides a thermal three-dimensional mapping method and device, a server and a readable storage medium, and the method comprises the steps: obtaining a three-dimensional model; wherein the three-dimensional model is generated according to image data acquired by the depth camera and the infrared camera in the same scene; obtaining heat map data; wherein the heat map data is data collected by the infrared camera and the depth camera in the same scene; aligning the three-dimensional model and the heat map data; and mapping the heat map data to the three-dimensional model. Visibly, the temperature distribution conditions of different areas in the three-dimensional scene are provided, and when the heat map is mapped to the surface of the three-dimensional model, a user can visually see the change of the temperature on the surface of the model, so that the temperature distribution conditions are better understood and analyzed.

Description

Technical Field

[0001] The present application belongs to the field of three-dimensional mapping technology, and in particular, relates to a thermal three-dimensional mapping method, device, server and readable storage medium. Background Art

[0002] Thermal imaging is a method that uses infrared technology to measure and display the temperature distribution of an object's surface. It is based on the principle that an object emits infrared radiation. It uses an infrared camera or thermal imager to capture the infrared radiation heat map of the target object and convert it into a visual image. However, thermal maps can only be displayed in two dimensions, and there is a lack of a three-dimensional thermal map display method that allows users to view the temperature distribution more intuitively. Summary of the invention

[0003] The embodiments of the present application provide a three-dimensional thermal mapping method, device, server and readable storage medium, which can solve the technical problem that there is a lack of a three-dimensional thermal map display method in the prior art, so that users can view the temperature distribution more intuitively.

[0004] In a first aspect, an embodiment of the present application provides a thermal three-dimensional mapping method, comprising: Acquire a three-dimensional model; wherein the three-dimensional model is generated based on image data collected by a depth camera and an infrared camera in the same scene; Obtain thermal map data; thermal map data is data collected by the infrared camera and the depth camera in the same scene; aligning the three-dimensional model and the thermal map data; The heat map data is mapped to the three-dimensional model.

[0005] In a possible implementation manner of the first aspect, obtaining a three-dimensional model includes: Get point cloud; The geometric structure is restored according to the point cloud to obtain the 3D model to be processed.

[0006] In a possible implementation manner of the first aspect, obtaining a point cloud includes: Get color image; Acquire a depth image corresponding to the color image; The point cloud is generated according to the color image and the depth image.

[0007] In a possible implementation manner of the first aspect, obtaining a point cloud includes: before aligning the three-dimensional model and the heat map data, further including: The three-dimensional model and the thermal map data are calibrated.

[0008] In a second aspect, an embodiment of the present application provides a thermal three-dimensional mapping device, comprising: A first acquisition module is used to acquire a three-dimensional model; wherein the three-dimensional model is generated based on image data collected by a depth camera and an infrared camera in the same scene; The second acquisition module is used to acquire thermal map data; wherein the thermal map data is data collected by the infrared camera and the depth camera in the same scene; An alignment module, used for aligning the three-dimensional model and the heat map data; A mapping module is used to map the thermal map data to the three-dimensional model.

[0009] In a possible implementation manner of the second aspect, the first acquisition module includes: The first acquisition submodule is used to acquire point clouds; The geometric structure recovery submodule is used to recover the geometric structure based on the point cloud to obtain the three-dimensional model to be processed.

[0010] In a possible implementation manner of the second aspect, the second acquisition module includes: The second acquisition submodule is used to acquire a color image; A third acquisition submodule is used for acquiring a depth image corresponding to the color image; A generating submodule is used to generate the point cloud according to the color image and the depth image.

[0011] In a possible implementation manner of the second aspect, the apparatus further includes: A calibration module is used to calibrate the three-dimensional model to be processed and the thermal map data to be processed.

[0012] In a third aspect, an embodiment of the present application provides a server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0014] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In the embodiment of the present application, a three-dimensional model is obtained; wherein the three-dimensional model is generated based on image data collected by a depth camera and an infrared camera in the same scene; thermal map data is obtained; wherein the thermal map data is data collected by the infrared camera and the depth camera in the same scene; the three-dimensional model and the thermal map data are aligned; and the thermal map data is mapped to the three-dimensional model. It can be seen that the embodiment of the present application provides the temperature distribution of different areas in the three-dimensional scene. When the thermal map is mapped to the surface of the three-dimensional model, the user can intuitively see the temperature change on the surface of the model, so as to better understand and analyze the temperature distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 It is a schematic diagram of the process of the thermal three-dimensional mapping method provided in the embodiment of the present application; Figure 2 is a structural block diagram of a thermal three-dimensional mapping device provided in an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. Implementation

[0017] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0018] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0019] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0020] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0021] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0022] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0023] The technical solution provided by the embodiments of the present application will be introduced below through specific examples.

[0024] See also Figure 1 , is a flow chart of a thermal three-dimensional mapping method provided in an embodiment of the present application. As an example but not a limitation, the method can be applied to a server. The method may include the following steps: Step S101, obtaining a three-dimensional model.

[0025] Among them, the three-dimensional model is generated based on the image data collected by the depth camera and the infrared camera in the same scene.

[0026] In specific applications, obtaining a three-dimensional model includes: Step S201, obtaining point cloud.

[0027] Exemplarily, obtaining a point cloud includes: Step S301, obtaining a color image.

[0028] Step S302: Acquire a depth image corresponding to the color image.

[0029] Step S303: generating a point cloud according to the color image and the depth image.

[0030] It is understandable that the depth camera can collect color images and depth images at the same time. Exemplarily, generating a point cloud based on the color image and the depth image includes: aligning the depth image and the color image according to an iterative closest point algorithm (ICP) so that the depth image and the color image have a unified coordinate system, and obtaining the three-dimensional coordinates of the point cloud according to the following formula: Among them, (u, v) is the pixel coordinate of each feature point in the color image, d is the depth value corresponding to each pixel in the color image, K is the intrinsic parameter of the depth camera, (X, Y, Z) is the three-dimensional coordinate of the point cloud, and the intrinsic parameter of the depth camera can be calculated using the Zhang Zhengyou calibration method.

[0031] Step S202: restore the geometric structure according to the point cloud to obtain a three-dimensional model to be processed.

[0032] Exemplarily, surface triangles of the point cloud are extracted according to a preset dense point cloud reconstruction MVS algorithm to generate a target three-dimensional model.

[0033] Step S102, obtaining thermal map data; wherein the thermal map data is data collected by the infrared camera and the depth camera in the same scene.

[0034] Step S103, aligning the three-dimensional model and the heat map data.

[0035] In an optional implementation, before aligning the three-dimensional model and the heat map data, the method further includes: Calibrate the 3D model and heat map data.

[0036] In specific applications, the 3D model and thermal map data are calibrated, including: (1) Use a calibration object: Place a calibration object in the scene. This object should be clearly visible in both the heat map and the 3D model and provide accurate spatial position and size information. For example, place a checkerboard with special textures or feature points in the scene.

[0037] (2) Collect calibration data: Use an infrared camera to shoot the calibration object in the same scene to obtain the corresponding thermal map data. At the same time, use a depth camera to scan the calibration object to obtain the real-scene 3D model data.

[0038] (3) Calculate the transformation matrix: By matching feature points, find the corresponding feature points in the heat map data and the 3D model data. Using these feature points, the transformation matrix between the heat map and the model can be calculated, such as the rigid body transformation matrix or the projection matrix.

[0039] Exemplarily, the process of calculating the transformation matrix is: fitting a 4×4 homogeneous transformation matrix using the least square method according to the projection transformation, representing translation, rotation, scaling and projection.

[0040] Specifically, aligning the three-dimensional model and the heat map data includes: realizing alignment between the coordinate system of the three-dimensional model and the coordinate system of the heat map data according to the calculated transformation matrix.

[0041] Step S104, mapping the heat map data to the three-dimensional model.

[0042] Exemplarily, the heat map data is linearly transformed from the original coordinate system to the coordinate system of the three-dimensional model according to the transformation matrix, so as to map the heat map data to the three-dimensional model.

[0043] In the embodiment of the present application, by obtaining a three-dimensional model; wherein the three-dimensional model is generated based on image data collected by a depth camera and an infrared camera in the same scene; obtaining thermal map data; wherein the thermal map data is data collected by the infrared camera and the depth camera in the same scene; aligning the three-dimensional model and the thermal map data; and mapping the thermal map data to the three-dimensional model. It can be seen that the embodiment of the present application provides the temperature distribution of different areas in the three-dimensional scene. When the thermal map is mapped to the surface of the three-dimensional model, the user can intuitively see the temperature changes on the surface of the model, so as to better understand and analyze the temperature distribution.

[0044] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0045] Corresponding to the thermal three-dimensional mapping method described in the above embodiment, Figure 2 A structural block diagram of a thermal three-dimensional mapping device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0046] Reference Figure 2 , the device comprises: A first acquisition module 21 is used to acquire a three-dimensional model; wherein the three-dimensional model is generated based on image data collected by a depth camera and an infrared camera in the same scene; The second acquisition module 22 is used to acquire thermal map data; wherein the thermal map data is data collected by the infrared camera and the depth camera in the same scene; An alignment module 23, used for aligning the three-dimensional model and the thermal map data; The mapping module 24 is used to map the thermal map data to the three-dimensional model.

[0047] In a possible implementation, the first acquisition module includes: The first acquisition submodule is used to acquire point clouds; The geometric structure recovery submodule is used to recover the geometric structure based on the point cloud to obtain the three-dimensional model to be processed.

[0048] In a possible implementation, the second acquisition module includes: The second acquisition submodule is used to acquire a color image; A third acquisition submodule is used for acquiring a depth image corresponding to the color image; A generating submodule is used to generate the point cloud according to the color image and the depth image.

[0049] In a possible implementation, the device further includes: A calibration module is used to calibrate the three-dimensional model to be processed and the thermal map data to be processed.

[0050] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0051] Figure 3 This is a schematic diagram of the structure of the server provided in the embodiment of the present application. Figure 3 As shown, the server 3 of this embodiment includes: at least one processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the at least one processor 30, and when the processor 30 executes the computer program 32, the steps in any of the above-mentioned method embodiments are implemented.

[0052] The server 3 may be a computing device such as a cloud server. The server may include, but is not limited to, a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of server 3 and does not constitute a limitation on server 3. Server 3 may include more or fewer components than shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0053] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0054] In some embodiments, the memory 31 may be an internal storage unit of the server 3, such as a hard disk or memory of the server 3. In other embodiments, the memory 31 may also be an external storage device of the server 3, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server 3. Further, the memory 31 may also include both an internal storage unit of the server 3 and an external storage device. The memory 31 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0055] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0056] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0057] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0058] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0059] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0060] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0061] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0062] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A thermal three-dimensional mapping method, characterized in that: include: Acquire a three-dimensional model; wherein the three-dimensional model is generated based on image data collected by a depth camera and an infrared camera in the same scene; Obtain thermal map data; thermal map data is data collected by the infrared camera and the depth camera in the same scene; aligning the three-dimensional model and the thermal map data; The heat map data is mapped to the three-dimensional model.

2. The thermal three-dimensional mapping method according to claim 1, characterized in that: Get 3D models, including: Get point cloud; The geometric structure is restored according to the point cloud to obtain the 3D model to be processed.

3. The thermal three-dimensional mapping method according to claim 2, characterized in that: Get point cloud, including: Get color image; Acquire a depth image corresponding to the color image; The point cloud is generated according to the color image and the depth image.

4. The thermal three-dimensional mapping method according to any one of claims 1 to 3, characterized in that: Before aligning the three-dimensional model and the heat map data, the method further includes: The three-dimensional model and the thermal map data are calibrated.

5. A thermal three-dimensional mapping device, characterized in that: include: A first acquisition module is used to acquire a three-dimensional model; wherein the three-dimensional model is generated based on image data collected by a depth camera and an infrared camera in the same scene; The second acquisition module is used to acquire thermal map data; wherein the thermal map data is data collected by the infrared camera and the depth camera in the same scene; An alignment module, used for aligning the three-dimensional model and the heat map data; A mapping module is used to map the thermal map data to the three-dimensional model.

6. The thermal three-dimensional mapping device according to claim 5, characterized in that: The first acquisition module includes: The first acquisition submodule is used to acquire point clouds; The geometric structure recovery submodule is used to recover the geometric structure based on the point cloud to obtain the three-dimensional model to be processed.

7. The thermal three-dimensional mapping device according to claim 6, characterized in that: The second acquisition module includes: The second acquisition submodule is used to acquire a color image; A third acquisition submodule is used for acquiring a depth image corresponding to the color image; A generating submodule is used to generate the point cloud according to the color image and the depth image.

8. The thermal three-dimensional mapping device according to any one of claims 5 to 7, characterized in that: The device further comprises: A calibration module is used to calibrate the three-dimensional model to be processed and the thermal map data to be processed.

9. A server comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images