Thermal imaging generation method and device of intelligent wearable equipment, equipment and storage medium

Abstract

The invention provides a thermal imaging generation method and device of intelligent wearable equipment, equipment and a storage medium, and the method comprises the steps: collecting a first image through an image sensor, carrying out the processing of the first image through an image processing algorithm, generating a second image, carrying out the feature recognition of the second image through a pre-trained data labeling model, and obtaining a thermal imaging result. The to-be-segmented region is obtained, so that the recognition efficiency and accuracy are improved; the method comprises the following steps: carrying out infrared spectrum IR segmentation marking on a to-be-segmented region, generating an IR numerical distribution diagram, converting an IR numerical value into a chromatic value by calibrating a proportionality coefficient, and carrying out coloring processing, thereby realizing thermal imaging processing on an image acquired by an image sensor, generating an equivalent thermal imaging image, avoiding integration of hardware equipment such as a thermal imager and the like, and improving the efficiency. The generation mode of the thermal imaging image is simplified, and the convenience and practicability of the intelligent wearable device in thermal imaging application are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for generating thermal images of a smart wearable device. Background Art

[0002] Smart wearable devices (such as AR glasses) cleverly integrate virtual information with the real world through optical display solutions. Thermal imagers can convert weak light or infrared rays into images visible to the naked eye at night or in low-light environments. Combining thermal imagers with smart wearable devices can improve the applicability of smart wearable devices and expand their application areas.

[0003] However, due to the complexity of existing infrared thermal imager solutions, they are costly and bulky, difficult to integrate into glasses, and inconvenient to carry, making the thermal imaging function of smart wearable devices difficult to achieve.

[0004] Therefore, how to improve the convenience of thermal imaging of smart wearable devices has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present application provides a thermal imaging generation method, device, equipment and storage medium for a smart wearable device, aiming to improve the convenience of thermal imaging of the smart wearable device.

[0006] In a first aspect, the present application provides a method for generating thermal imaging of a smart wearable device, the method comprising:

[0007] Based on the image sensor in the smart wearable device, at least one frame of the first image is collected;

[0008] Based on an image processing algorithm, performing image processing on the first image to generate a second image;

[0009] Based on the pre-trained data annotation model, feature recognition is performed on the second image to obtain a region to be segmented corresponding to at least one identified object;

[0010] Performing infrared spectrum IR segmentation and marking on at least one of the to-be-segmented regions to generate an IR value distribution map;

[0011] Based on the calibration ratio coefficient between the IR component and the chromaticity component, the IR values ​​in the IR value distribution map are converted into chromaticity values, and colorization is performed according to the chromaticity values ​​to generate a thermal imaging image.

[0012] In a second aspect, the present application further provides a thermal imaging generation device for a smart wearable device, the thermal imaging generation device for the smart wearable device comprising:

[0013] A first image acquisition module, configured to acquire at least one frame of a first image based on an image sensor in the smart wearable device;

[0014] A first image processing module, configured to perform image processing on the first image based on an image processing algorithm to generate a second image;

[0015] An image feature recognition module, used to perform feature recognition on the second image based on a pre-trained data annotation model to obtain a region to be segmented corresponding to at least one recognized object;

[0016] An infrared spectrum marking module, used for performing infrared spectrum IR segmentation marking on at least one of the to-be-segmented regions to generate an IR value distribution map;

[0017] The thermal imaging image generation module is used to convert the IR values ​​in the IR value distribution map into chromaticity values ​​based on the calibration ratio coefficient between the IR component and the chromaticity component, and perform coloring processing according to the chromaticity value to generate a thermal imaging image.

[0018] In a third aspect, the present application also provides a smart wearable device, which includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the thermal imaging generation method of the smart wearable device as described above are implemented.

[0019] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the thermal imaging generation method for the smart wearable device as described above are implemented.

[0020] In a fifth aspect, the present application also provides an AR glasses, comprising a camera, a sensor, a processor and a computer-readable storage medium storing a computer program, wherein the camera and the sensor are arranged on the frame of the AR glasses, and when the computer program is executed by the processor, the steps of the thermal imaging generation method of the smart wearable device as described above are implemented.

[0021] The present application provides a method, device, smart wearable device and storage medium for generating thermal imaging of a smart wearable device. The method of the present application collects a first image through an image sensor, and uses an image processing algorithm to process the first image to generate a second image, thereby improving the clarity and quality of the image and making subsequent feature recognition more accurate. The pre-trained data annotation model is used to perform feature recognition on the second image to obtain the area to be segmented, thereby improving the efficiency and accuracy of recognition; the infrared spectrum IR segmentation and annotation of the area to be segmented is performed to generate an IR numerical distribution map, which provides accurate temperature distribution data for subsequent thermal imaging, and the IR numerical value is converted into a chromaticity value and colorized to generate a thermal imaging image. The calibration ratio coefficient between the IR component and the chromaticity component is calculated by image processing technology, and the visible light image is converted into a thermal imaging image by the calibration ratio coefficient, so as to realize the thermal imaging processing of the image collected by the image sensor, generate an equivalent thermal imaging image, avoid the integration of hardware devices such as thermal imagers, simplify the generation method of thermal imaging images, and improve the convenience and practicality of smart wearable devices in thermal imaging applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 A schematic diagram of a flow chart of a first embodiment of a thermal imaging generation method for a smart wearable device provided in an embodiment of the present application;

[0024] Figure 2 A schematic diagram of object recognition and selection results of a first image provided in an embodiment of the present application;

[0025] Figure 3 IR value distribution diagram provided for the embodiments of the present application;

[0026] Figure 4 A thermal imaging calibration image corresponding to the first image provided in the embodiment of the present application;

[0027] Figure 5 It is a structural schematic diagram of a first embodiment of a thermal imaging generation device for a smart wearable device provided by the present application;

[0028] Figure 6 It is a schematic block diagram of the structure of a smart wearable device provided in an embodiment of the present application.

[0029] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0032] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0033] Please refer to Figure 1 , Figure 1 A schematic flow chart of a first embodiment of a thermal imaging generation method for a smart wearable device provided in an embodiment of the present application.

[0034] like Figure 1 As shown, the thermal imaging generation method of the smart wearable device includes steps S101 to S105.

[0035] S101, collecting at least one frame of a first image based on an image sensor in a smart wearable device;

[0036] In one embodiment, the smart wearable device provided in the present application may be AR glasses, VR glasses, smart helmets and other devices. At least one image sensor may be configured in the smart wearable device to capture environmental images around the user of the smart wearable device.

[0037] In one embodiment, the image sensor may be a sensor with a wavelength response of 400-1500nm, and a layer of IR (Infrared) coating with a response range of 400-1500nm is coated on the surface of the image sensor, wherein the visible light portion is used for normal image acquisition, and the infrared portion is used to collect IR data reflected by objects in the scene.

[0038] Applying an IR coating with a response range of 400-1500nm on the surface of the image sensor can enhance the response of the image sensor to infrared light, which not only improves the sensitivity of the sensor in the infrared region, but also protects the image sensor from environmental factors.

[0039] In an exemplary embodiment, the number of pixels of the image sensor can be less than or equal to 5M (2592*1944), the pixel size can be large, such as about 3um, and the sensor can have HDR technology such as DOL. The image sensor with a large pixel size has a large amount of light intake and can be well integrated with the HDR circuit. Because thermal imaging is required, it is necessary to ensure that the collected IR light is not overexposed; and because the pixel size is selected to be large, the number of pixels cannot be too large, otherwise the image size will be too large.

[0040] Generally speaking, as long as the temperature of an object is higher than absolute zero, it will emit a certain amount of infrared radiation. Therefore, smart wearable devices can receive the infrared radiation energy emitted by objects through the camera of the SWIR (Short Wavelength Infra-Red) sensor. In addition, the camera of the SWIR camera itself can also form an image. The two can be processed and combined in post-processing to generate a thermal image.

[0041] S102, performing image processing on the first image based on an image processing algorithm to generate a second image;

[0042] In one embodiment, an image signal processor ISP (Image Signal Processor) can be configured in the smart wearable device. After the image sensor captures a first image, the first image is input into the ISP. The image processing algorithm in the ISP is used to perform image processing on the first image, including denoising, exposure control, white balance, color correction, sharpening, contrast enhancement, lens distortion correction, bad pixel correction, color space conversion, dynamic range compression, multi-frame noise reduction and other image operations.

[0043] Among them, ISP (Image Signal Processor) is an image signal processor, which is a chip or integrated circuit module specially used to process digital image signals. After the image sensor (such as CMOS or CCD) captures the light, it converts the light signal into an electrical signal in RAW format. These RAW data contain unprocessed image information, such as brightness, color, etc., but usually have problems such as noise and color deviation. The ISP processor can perform black level correction, white balance correction, color correction, sharpening and brightness optimization on the RAW image.

[0044] Specifically, the image sensor first captures the raw data of the image, which is usually RAW image data with noise, lack of color information and low contrast, i.e. the first image. The raw image data can undergo preliminary data preprocessing steps, such as signal amplification, analog-to-digital conversion, etc., in preparation for the subsequent image processing stage. The ISP performs complex processing on the raw data, including denoising, color correction, demosaicing, dynamic range optimization, sharpening, etc., and finally obtains an image with optimized visual effects, i.e. the second image.

[0045] S103, performing feature recognition on the second image based on a pre-trained data annotation model to obtain a to-be-segmented region corresponding to at least one recognized object;

[0046] Generally speaking, data labeling refers to the process of converting raw data into machine-readable form, including classifying, labeling, processing and cleaning data.

[0047] In one embodiment, the first image is processed by ISP into a second image, and the second image is input into a pre-trained data annotation model. The data annotation model identifies and annotates the classification categories of multiple objects in the second image: such as people, animals, flowers, trees, sky, water, houses, etc., and distinguishes and classifies them in the form of outline selection. The data annotation model selects the objects in the second image as follows: Figure 2 shown.

[0048] Specifically, the image sensor can use a sensor with a wavelength response of 400-1500nm, covering the spectral range from visible light to near infrared (NIR). Among them, the visible light part (400-700nm) is the light wave range that the human eye can perceive and is used for normal image acquisition; within this range, the sensor can capture the colors and details that the human eye sees in daily life. The light in the near infrared part (700-1500nm) is invisible and is used to collect the amount of infrared data reflected by objects in the scene.

[0049] In one embodiment, visible light imaging is used to facilitate the identification and collection of objects in the imaging scene, classify the objects, and extract their contours, thereby finely drawing a thermal image corresponding to the imaging scene, thereby improving the drawing accuracy of the thermal image of the imaging scene.

[0050] Furthermore, based on the data annotation model, image features of the second image are identified, and at least one type of identification object and the image contour of at least one identification object in the second image are determined; based on the object type and the image contour of at least one identification object, at least one area to be segmented is determined.

[0051] In one embodiment, the data annotation model performs feature recognition on the second image, identifies identifiable objects in the second image, such as people, animals, flowers, trees, buildings, vehicles, sky, ground, etc., and determines the type of the identified object, such as animals, plants, tools, buildings, natural scenery (such as the sky, etc.), etc. Among them, the identified objects and the classification of the identified objects can be set according to actual needs.

[0052] In one embodiment, the data annotation model can distinguish and classify different types of recognition objects by means of outline selection, etc. For example, the outlines of objects such as people and animals can be selected by using different colors or lines.

[0053] In one embodiment, based on the image contours of each recognized object, the data annotation model divides the second image into areas to be segmented. For example, the building in the second image is used as one area to be segmented, the sky is used as a second area to be segmented, and the person is used as a third area to be segmented, and so on. The second image is divided into multiple areas to be segmented for further processing or analysis.

[0054] Furthermore, based on the object type of the identified object and the image contour, the pixel size of the image contour corresponding to the identified object of the same object type is determined; when the pixel size is greater than or equal to a preset size threshold, the identified object of the same object type is segmented to obtain the area to be segmented corresponding to the identified object.

[0055] First, determine whether the pixel size of the image contour corresponding to each type of recognition object is greater than a preset size threshold, such as a 300*200 pixel area. If the pixel size of the image contour corresponding to the same type of recognition object is greater than or equal to the preset size threshold, the image contour of the recognition object participates in the segmentation statistics, that is, the image contour of the recognition object is taken as a region to be segmented.

[0056] In one embodiment, when the pixel size is smaller than the preset size threshold, the image contour corresponding to the identified object is merged with the image contour of its adjacent identified object to obtain the merged image contour; when the merged image contour is greater than or equal to the preset size threshold, the merged image contour is segmented to obtain the area to be segmented.

[0057] If the pixel size of the image contour corresponding to the same type of identification object is less than the preset size threshold, the image contour of the identification object and its adjacent identification object are merged and calculated. If the pixel size of the merged image contour is greater than or equal to the preset size threshold, the merged image contour is divided into a region to be segmented.

[0058] If the pixel size of the merged image contour is still smaller than the preset size threshold, the merged image contour will continue to be merged with the image contours of other adjacent recognition objects and participate in the segmentation statistics until the merged image contour is greater than or equal to the preset size threshold, and the merged image contour will be divided into a region to be segmented.

[0059] In another embodiment, if there are identification objects of the same type in different regions of the second image and they are not adjacent, the image contours of the different regions of the identification objects of the same type may be segmented and counted separately.

[0060] For example, if there are only two human objects in the second image, and the distribution positions of the two human objects are relatively separated, they can be segmented and counted separately. That is, if the image contour sizes corresponding to the two human objects are greater than or equal to the preset size threshold, they can be divided into two areas to be segmented respectively; if the image contour sizes of the two human objects are less than the preset size threshold, the image contours of the two human objects and their adjacent identification objects can be merged respectively and then participate in the segmentation statistics.

[0061] In another embodiment, if the same type of identification objects are located in different areas of the second image and are not completely adjacent to each other, but are distributed in a relatively concentrated position, they can be merged and regarded as the same image contour.

[0062] For example, if there are multiple human objects in the second image, most of which are concentrated in a certain area of ​​the second image, these human objects can be divided into the same image contour, but there are still a few, such as one or two human objects, distributed at a position far away from the concentrated area. At this time, the few human objects can be segmented and counted separately. If the image contour size corresponding to the few human objects is greater than or equal to the preset size threshold, the image contour corresponding to the few human objects can be divided into a to-be-segmented area; if the image contour size corresponding to the few human objects is less than the preset size threshold, the image contour corresponding to the few human objects can be merged with its adjacent image contours and then participate in the segmentation statistics.

[0063] In one embodiment, when merging image contours, if there are multiple adjacent image contours to be merged, the image contour to be merged may be preferentially merged with an adjacent image contour having a smaller image contour pixel size.

[0064] It is understandable that the image contour described in the embodiment of the present application is only for ease of understanding and is used to express relevant concepts, and does not constitute a limitation on the scope of protection. The image contour can be other files containing image information obtained by image processing of the part of the image corresponding to the identified object, or it can be the part of the image corresponding to the identified object itself.

[0065] S104, performing infrared spectrum IR segmentation and annotation on at least one of the to-be-segmented regions to generate an IR value distribution map;

[0066] In one embodiment, the various types of identified objects are classified and divided into a plurality of to-be-segmented regions according to their types, and an IR component is marked on each to-be-segmented region.

[0067] Furthermore, based on a preset segmentation size, each of the to-be-segmented areas is segmented to obtain at least one annotated area; based on the type of the identified object corresponding to at least one of the annotated areas, the IR value of each annotated area is annotated to generate the IR value distribution map.

[0068] In one embodiment, after the regions to be segmented are divided, each region to be segmented is segmented according to a preset segmentation size (such as 50*50 pixels), and the IR values ​​of each region to be segmented are counted, and the following is generated: Figure 3 IR value distribution diagram shown.

[0069] Specifically, each area to be segmented is distinguished and labeled according to the type of identification object in the area to be segmented and the infrared spectrum characteristics, that is, according to the different characteristics exhibited by each type of identification object under the infrared spectrum, the infrared spectrum value (IR component) of the area to be segmented corresponding to each identification object is labeled.

[0070] In one embodiment, in a thermal imaging image, different infrared spectrum values ​​appear in different colors, for example, a hotter area may appear in red or yellow, and a colder area may appear in blue or green, etc. The numerical conversion ratio between the IR component and the chromaticity component can be calibrated using a real thermal image captured by a thermal imager.

[0071] Furthermore, based on a thermal imager, scene images of the image sensor in the same scene are collected to obtain a thermal imaging calibration image corresponding to the first image; based on a thermal imaging image processing algorithm, chromaticity values ​​are annotated on the thermal imaging calibration image to obtain a chromaticity value calibration image corresponding to the first image; based on the correspondence between the IR value distribution diagram and the chromaticity value calibration image, a calibration ratio coefficient between the IR component and the chromaticity component is calculated.

[0072] In one embodiment, a thermal image of an imaging scene is collected under the same acquisition conditions of an image sensor in a smart wearable device by a standard thermal imager. During the calibration process, calibration can be performed under a standard fixed environment, and the thermal imager and the image sensor are set to the same image acquisition parameters, including internal parameters (such as focal length, pixel size, principal point coordinates, etc.), external parameters (such as rotation matrix and translation vector and other posture parameters) and consistency of the external environment. For example, the consistency of ambient temperature and humidity, the subject can use a reference source of known temperature (such as a black body, etc.), and different temperature points can be used for calibration, so that the thermal imager and the image sensor can collect images of the same calibration scene to improve the accuracy of the calibration scale coefficient.

[0073] In one embodiment, after the thermal imager captures the image sensor, Figure 4 After obtaining the thermal imaging calibration image corresponding to the first image shown, the thermal imaging calibration image is annotated with chromaticity values ​​based on the thermal imaging image processing algorithm. Generally, the color space conversion of the image is adopted, for example, from the RGB color space to the HLS color space, and the range of the specific color component is extracted. All images in the red range are changed to standard red, and other colors are changed to standard blue, and then converted back to the RGB color space to obtain an accurate chromaticity value calibration image. After converting back to the RGB color space, the processed thermal imaging calibration image can be converted from the RGB color space to the grayscale space, and histogram equalization and filtering processing can be performed to improve the image quality.

[0074] In one embodiment, by analyzing Figure 4 exist Figure 3 The chromaticity value of the marked area is obtained as a chromaticity value M(x, y) and IR data. Since the chromaticity value changes slightly, it needs to be multiplied by a coefficient K, namely KM, to enlarge the range of chromaticity value changes, and the calibration ratio coefficient is obtained as Ki(IR / KM).

[0075] For example, based on the correspondence between the IR value distribution diagram and the chromaticity value calibration image, a mathematical model representing the linear or nonlinear relationship between the IR component and the chromaticity component can be established, such as analyzing the image pixel values ​​through polynomial fitting or a neural network model, and calculating the model parameters, i.e., the calibration scale coefficient, through the least squares method or other optimization algorithms, thereby determining the relationship between different temperatures and chromaticity values.

[0076] In one embodiment, based on the thermal imager and the image sensor, a calibrated thermal imaging image and a visible light imaging image in at least one calibration scene are collected; the chromaticity values ​​in the calibrated thermal imaging image are marked, and the IR values ​​of the visible light imaging image are marked; based on the calibrated thermal imaging image and the visible light imaging image corresponding to at least one of the calibration scenes, an initial calibration proportional coefficient between the IR component and the chromaticity component in at least one of the calibration scenes is calculated; the initial calibration proportional coefficient corresponding to each of the calibration scenes is fitted to determine the calibration proportional coefficient.

[0077] In one embodiment, in the same region or in two or more regions with a short spatial distance, multiple environments in the region can be calibrated to generate initial calibration scale coefficient Ki values ​​under different environments, and an accurate scale coefficient K can be obtained by fitting as a universal calibration scale coefficient.

[0078] In one embodiment, in different regions or two or more regions that are far apart in space, factors such as climate, topography, and air humidity may affect the refraction and scattering of light, thereby affecting the IR value acquired by the image sensor. Therefore, multiple environments in different regions may be calibrated separately, and initial calibration proportional coefficient Ki values ​​may be generated for each environment, and a proportional coefficient K may be obtained through fitting as the corresponding calibration proportional coefficient.

[0079] Specifically, the chromaticity values ​​in the collected calibration thermal imaging images can be annotated by image processing techniques (such as image segmentation and color recognition algorithms). By digital processing of the image, the pixel values ​​of the visible light imaging image are converted into corresponding temperature values, and the IR values ​​of the collected visible light imaging image are annotated. The conversion relationship between the chromaticity values ​​in the calibration thermal imaging image and the IR values ​​in the visible light imaging image under the same calibration scene, that is, the initial calibration scale coefficient, is calculated by a mathematical model. This is achieved by nonlinear fitting techniques, such as polynomial fitting or neural network models, and statistical analysis and fitting of the initial calibration scale coefficient Ki in each scene are performed to find the calibration scale coefficient that best suits all data.

[0080] S105 , based on the calibration ratio coefficient between the IR component and the chromaticity component, convert the IR values ​​in the IR value distribution diagram into chromaticity values, and perform coloring processing according to the chromaticity values ​​to generate a thermal imaging image.

[0081] In one embodiment, according to the pre-determined calibration ratio coefficient between the IR component and the chromaticity component, the IR value in the IR value distribution diagram is converted into the chromaticity value, thereby converting the IR value distribution diagram into the chromaticity distribution diagram, and the color diagram distribution diagram is colored according to the chromaticity value corresponding to each area, so as to obtain a thermal imaging image.

[0082] Specifically, the IR value of each pixel in the IR value distribution map can be converted into a corresponding chromaticity value through a linear or nonlinear mapping function according to the calibration scale coefficient. For example, if it is known that the corresponding relationship between the IR value and the chromaticity value is linear, a linear formula can be used for conversion: chromaticity value = k × IR value + b. Among them, k is the calibration scale coefficient, and b is the intercept value, which can be obtained through the calibration process.

[0083] For example, the converted chromaticity values ​​may be normalized to fit the chromaticity range of the display device. For example, if the chromaticity value range is from 0 to 255, then the normalization may be performed using the following formula:

[0084]

[0085] In one embodiment, the chromaticity values ​​are mapped into the RGB color space, and then the calculated RGB values ​​are combined into a final thermal imaging image.

[0086] This embodiment provides a thermal imaging generation method for a smart wearable device, which collects a first image through an image sensor, and uses an image processing algorithm to process the first image to generate a second image, thereby improving the clarity and quality of the image and making subsequent feature recognition more accurate. The pre-trained data annotation model is used to perform feature recognition on the second image to obtain the area to be segmented, thereby improving the efficiency and accuracy of recognition; the infrared spectrum IR segmentation and annotation of the area to be segmented is performed to generate an IR value distribution map, which provides accurate temperature distribution data for subsequent thermal imaging, and the IR value is converted into a chromaticity value and colorized to generate a thermal imaging image. The calibration ratio coefficient between the IR component and the chromaticity component is calculated by image processing technology, and the visible light image is converted into a thermal imaging image by the calibration ratio coefficient, so as to realize thermal imaging processing of the image collected by the image sensor, generate an equivalent thermal imaging image, avoid the integration of hardware devices such as thermal imagers, simplify the generation method of thermal imaging images, and improve the convenience and practicality of smart wearable devices in thermal imaging applications.

[0087] In one embodiment, the embodiment of the present application provides an AR glasses, including a camera, a sensor, a processor and a computer-readable storage medium storing a computer program, wherein the camera and the sensor are arranged on the frame of the AR glasses, and when the computer program is executed by the processor, the thermal imaging generation method of the above smart wearable device is implemented.

[0088] See also Figure 5 , Figure 5 It is a structural schematic diagram of a first embodiment of a thermal imaging generation device for a smart wearable device provided in the present application, and the thermal imaging generation device for the smart wearable device is used to execute the aforementioned thermal imaging generation method for the smart wearable device.

[0089] like Figure 5 As shown, the thermal imaging generation device 200 of the smart wearable device includes: a first image acquisition module 201, a first image processing module 202, an image feature recognition module 203, an infrared spectrum annotation module 204 and a thermal imaging image generation module 205.

[0090] The first image acquisition module 201 is used to acquire at least one frame of the first image based on the image sensor in the smart wearable device;

[0091] A first image processing module 202, configured to perform image processing on the first image based on an image processing algorithm to generate a second image;

[0092] An image feature recognition module 203 is used to perform feature recognition on the second image based on a pre-trained data annotation model to obtain a region to be segmented corresponding to at least one recognition object;

[0093] The infrared spectrum marking module 204 is used to perform infrared spectrum IR segmentation and marking on at least one of the to-be-segmented regions to generate an IR value distribution map;

[0094] The thermal imaging image generation module 205 is used to convert the IR values ​​in the IR value distribution diagram into chromaticity values ​​based on the calibration ratio coefficient between the IR component and the chromaticity component, and perform coloring processing according to the chromaticity values ​​to generate a thermal imaging image.

[0095] In one embodiment, the image feature recognition module 203 includes:

[0096] An image feature recognition unit, configured to recognize image features of the second image based on the data annotation model, and determine at least one type of recognized object and an image contour of at least one of the recognized objects in the second image;

[0097] The to-be-segmented region determining unit is used to determine at least one of the to-be-segmented regions based on the object type of at least one of the identified objects and the image contour.

[0098] In one embodiment, the to-be-segmented area determining unit includes:

[0099] A pixel size determination subunit, configured to determine the pixel size of the image contour corresponding to the identified object of the same object type based on the object type of the identified object and the image contour;

[0100] The first subunit for determining the area to be segmented is used to segment the identified objects of the same object type when the pixel size is greater than or equal to a preset size threshold, to obtain the area to be segmented corresponding to the identified objects.

[0101] In one embodiment, the to-be-segmented area determining unit further includes:

[0102] An image contour merging subunit, configured to merge the image contour corresponding to the identified object with the image contour of an adjacent identified object to obtain the merged image contour when the pixel size is smaller than the preset size threshold;

[0103] The second subunit for determining the region to be segmented is used to segment the merged image contour when the merged image contour is greater than or equal to the preset size threshold to obtain the region to be segmented.

[0104] In one embodiment, the thermal imaging generation device 200 of the smart wearable device further includes a scale factor calibration module, including:

[0105] A thermal imaging acquisition unit, used to acquire a scene image of the image sensor in the same scene based on a thermal imager, and obtain a thermal imaging calibration image corresponding to the first image;

[0106] a chromaticity value labeling unit, configured to label the chromaticity value of the thermal imaging calibration image based on a thermal imaging image processing algorithm, so as to obtain a chromaticity value labeling image corresponding to the first image;

[0107] The calibration ratio coefficient calculation unit is used to calculate the calibration ratio coefficient between the IR component and the chromaticity component based on the corresponding relationship between the IR value distribution diagram and the chromaticity value calibration image.

[0108] In one embodiment, the proportional coefficient calibration module further includes:

[0109] An image acquisition unit, used for acquiring a calibration thermal imaging image and a visible light imaging image in at least one calibration scene based on a thermal imager and the image sensor;

[0110] A data annotation unit, used to annotate the chromaticity value in the calibration thermal imaging image, and annotate the IR value of the visible light imaging image;

[0111] A proportionality coefficient calculation unit, configured to calculate an initial calibration proportionality coefficient between the IR component and the chromaticity component in at least one calibration scene based on the calibration thermal imaging image and the visible light imaging image corresponding to at least one calibration scene;

[0112] The calibration scale coefficient fitting unit is used to fit the initial calibration scale coefficient corresponding to each calibration scene to determine the calibration scale coefficient.

[0113] In one embodiment, the infrared spectrum annotation module 204 includes:

[0114] A region segmentation unit, configured to segment each of the regions to be segmented based on a preset segmentation size to obtain at least one marked region;

[0115] The infrared value marking unit is used to mark the IR value of each marked area based on the type of the identification object corresponding to at least one of the marked areas, and generate the IR value distribution map.

[0116] It should be noted that technicians in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned thermal imaging generation method embodiment of the smart wearable device, and will not be repeated here.

[0117] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 6 The system runs on the smart wearable device shown.

[0118] See also Figure 6 , Figure 6 1 is a schematic block diagram of the structure of a smart wearable device provided in an embodiment of the present application. The smart wearable device may be a server.

[0119] See also Figure 6 The smart wearable device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0120] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any thermal imaging generation method of a smart wearable device.

[0121] The processor is used to provide computing and control capabilities to support the operation of the entire smart wearable device.

[0122] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any thermal imaging generation method of the smart wearable device.

[0123] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the smart wearable device to which the scheme of the present application is applied. The specific smart wearable device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0124] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0125] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0126] Based on the image sensor in the smart wearable device, at least one frame of the first image is collected;

[0127] Based on an image processing algorithm, performing image processing on the first image to generate a second image;

[0128] Based on the pre-trained data annotation model, feature recognition is performed on the second image to obtain a region to be segmented corresponding to at least one identified object;

[0129] Performing infrared spectrum IR segmentation and annotation on at least one of the to-be-segmented regions to generate an IR value distribution map;

[0130] Based on the calibration ratio coefficient between the IR component and the chromaticity component, the IR values ​​in the IR value distribution map are converted into chromaticity values, and colorization is performed according to the chromaticity values ​​to generate a thermal imaging image.

[0131] In one embodiment, when the processor implements the pre-trained data annotation model to perform feature recognition on the second image and obtains the to-be-segmented area corresponding to at least one recognized object, it is configured to implement:

[0132] Based on the data annotation model, identify image features of the second image, and determine at least one type of recognized object and at least one image contour of the recognized object in the second image;

[0133] At least one of the regions to be segmented is determined based on the object type of at least one of the identified objects and the image contour.

[0134] In one embodiment, when the processor determines at least one area to be segmented based on at least one object type of the identified object and the image contour, it is configured to implement:

[0135] Determining, based on the object type of the identified object and the image contour, a pixel size of the image contour corresponding to the identified object of the same object type;

[0136] When the pixel size is greater than or equal to a preset size threshold, the identified objects of the same object type are segmented to obtain the to-be-segmented areas corresponding to the identified objects.

[0137] In one embodiment, after determining the pixel size of the image contour corresponding to the identified object of the same object type based on the object type of the identified object and the image contour, the processor is further configured to implement:

[0138] When the pixel size is smaller than the preset size threshold, merging the image contour corresponding to the recognition object with the image contour of its adjacent recognition object to obtain the merged image contour;

[0139] When the merged image contour is greater than or equal to the preset size threshold, the merged image contour is segmented to obtain the area to be segmented.

[0140] In one embodiment, before implementing the calibration ratio coefficient between the IR component and the chromaticity component, converting the IR values ​​in the IR value distribution map into chromaticity values, and performing coloring according to the chromaticity values ​​to generate the thermal imaging image, the processor is further used to implement:

[0141] Based on a thermal imager, a scene image of the image sensor in the same scene is collected to obtain a thermal imaging calibration image corresponding to the first image;

[0142] Based on a thermal imaging image processing algorithm, the thermal imaging calibration image is annotated with chromaticity values ​​to obtain a chromaticity value calibration image corresponding to the first image;

[0143] Based on the corresponding relationship between the IR value distribution diagram and the chromaticity value calibration image, a calibration proportional coefficient between the IR component and the chromaticity component is calculated.

[0144] In one embodiment, before implementing the calibration ratio coefficient between the IR component and the chromaticity component, converting the IR values ​​in the IR value distribution map into chromaticity values, and performing coloring according to the chromaticity values ​​to generate the thermal imaging image, the processor is further used to implement:

[0145] Based on the thermal imager and the image sensor, collecting a calibration thermal imaging image and a visible light imaging image in at least one calibration scene;

[0146] marking the chromaticity value in the calibration thermal imaging image, and marking the IR value of the visible light imaging image;

[0147] Calculating an initial calibration proportional coefficient between the IR component and the chromaticity component in at least one of the calibration scenes based on the calibration thermal imaging image and the visible light imaging image corresponding to at least one of the calibration scenes;

[0148] The initial calibration scale coefficient corresponding to each calibration scene is fitted to determine the calibration scale coefficient.

[0149] In one embodiment, when the processor performs infrared spectrum IR segmentation and marking on at least one of the to-be-segmented regions and generates an IR value distribution map, it is configured to implement:

[0150] Based on a preset segmentation size, each of the to-be-segmented regions is segmented to obtain at least one marked region;

[0151] Based on the type of the identification object corresponding to at least one of the marked areas, the IR value of each of the marked areas is marked to generate the IR value distribution map.

[0152] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement a thermal imaging generation method for any smart wearable device provided in an embodiment of the present application.

[0153] The computer-readable storage medium may be an internal storage unit of the smart wearable device described in the foregoing embodiment, such as a hard disk or memory of the smart wearable device. The computer-readable storage medium may also be an external storage device of the smart wearable device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the smart wearable device.

[0154] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating thermal imaging of a smart wearable device, characterized in that: The method comprises: Based on the image sensor in the smart wearable device, at least one frame of the first image is collected; Based on an image processing algorithm, performing image processing on the first image to generate a second image; Based on the pre-trained data annotation model, feature recognition is performed on the second image to obtain a region to be segmented corresponding to at least one identified object; Performing infrared spectrum IR segmentation and annotation on at least one of the to-be-segmented regions to generate an IR value distribution map; Based on the calibration ratio coefficient between the IR component and the chromaticity component, the IR values ​​in the IR value distribution map are converted into chromaticity values, and colorization is performed according to the chromaticity values ​​to generate a thermal imaging image.

2. The thermal imaging generation method of the smart wearable device according to claim 1, characterized in that: The step of performing feature recognition on the second image based on the pre-trained data annotation model to obtain a region to be segmented corresponding to at least one recognized object includes: Based on the data annotation model, identify image features of the second image, and determine at least one type of recognized object and at least one image contour of the recognized object in the second image; At least one of the regions to be segmented is determined based on the object type of at least one of the identified objects and the image contour.

3. The thermal imaging generation method of the smart wearable device according to claim 2, characterized in that: The determining at least one region to be segmented based on the object type of at least one identified object and the image contour comprises: Determining, based on the object type of the identified object and the image contour, a pixel size of the image contour corresponding to the identified object of the same object type; When the pixel size is greater than or equal to a preset size threshold, the identified objects of the same object type are segmented to obtain the to-be-segmented areas corresponding to the identified objects.

4. The thermal imaging generation method of the smart wearable device according to claim 3, characterized in that: After determining the pixel size of the image contour corresponding to the identified object of the same object type based on the object type of the identified object and the image contour, the method further includes: When the pixel size is smaller than the preset size threshold, merging the image contour corresponding to the recognition object with the image contour of its adjacent recognition object to obtain the merged image contour; When the merged image contour is greater than or equal to the preset size threshold, the merged image contour is segmented to obtain the area to be segmented.

5. The thermal imaging generation method of the smart wearable device according to claim 1, characterized in that: The method further includes converting the IR values ​​in the IR value distribution diagram into chromaticity values ​​based on the calibration ratio coefficient between the IR component and the chromaticity component, and performing coloring according to the chromaticity values ​​to generate the thermal imaging image. Based on a thermal imager, a scene image of the image sensor in the same scene is collected to obtain a thermal imaging calibration image corresponding to the first image; Based on a thermal imaging image processing algorithm, the thermal imaging calibration image is annotated with chromaticity values ​​to obtain a chromaticity value calibration image corresponding to the first image; Based on the corresponding relationship between the IR value distribution diagram and the chromaticity value calibration image, a calibration proportional coefficient between the IR component and the chromaticity component is calculated.

6. The thermal imaging generation method of the smart wearable device according to claim 1, characterized in that: The method further includes converting the IR values ​​in the IR value distribution diagram into chromaticity values ​​based on the calibration ratio coefficient between the IR component and the chromaticity component, and performing coloring according to the chromaticity values ​​to generate the thermal imaging image. Based on the thermal imager and the image sensor, collecting a calibration thermal imaging image and a visible light imaging image in at least one calibration scene; marking the chromaticity value in the calibration thermal imaging image, and marking the IR value of the visible light imaging image; Calculating an initial calibration proportional coefficient between the IR component and the chromaticity component in at least one of the calibration scenes based on the calibration thermal imaging image and the visible light imaging image corresponding to at least one of the calibration scenes; The initial calibration scale coefficient corresponding to each calibration scene is fitted to determine the calibration scale coefficient.

7. The thermal imaging generation method of the smart wearable device according to claim 1, characterized in that: The step of performing infrared spectrum IR segmentation and marking on at least one of the to-be-segmented regions to generate an IR value distribution map includes: Based on a preset segmentation size, each of the to-be-segmented regions is segmented to obtain at least one marked region; Based on the type of the identification object corresponding to at least one of the marked areas, the IR value of each of the marked areas is marked to generate the IR value distribution map.

8. A thermal imaging generation device for a smart wearable device, characterized in that: The thermal imaging generation device of the smart wearable device comprises: A first image acquisition module, configured to acquire at least one frame of a first image based on an image sensor in the smart wearable device; A first image processing module, configured to perform image processing on the first image based on an image processing algorithm to generate a second image; An image feature recognition module, used to perform feature recognition on the second image based on a pre-trained data annotation model to obtain a region to be segmented corresponding to at least one recognized object; An infrared spectrum marking module, used for performing infrared spectrum IR segmentation marking on at least one of the to-be-segmented regions to generate an IR value distribution map; The thermal imaging image generation module is used to convert the IR values ​​in the IR value distribution map into chromaticity values ​​based on the calibration ratio coefficient between the IR component and the chromaticity component, and perform coloring processing according to the chromaticity value to generate a thermal imaging image.

9. A smart wearable device, characterized in that: The smart wearable device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the thermal imaging generation method of the smart wearable device according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the thermal imaging generation method of the smart wearable device according to any one of claims 1 to 7 are implemented.

11. An AR glasses, comprising a camera, a sensor, a processor, and a computer-readable storage medium storing a computer program, characterized in that: The camera and the sensor are arranged on the frame of the AR glasses, and when the computer program is executed by the processor, the steps of the thermal imaging generation method of the smart wearable device according to any one of claims 1 to 7 are implemented.

Images