Target area detection method and device, model training method and device, equipment and medium

Abstract

The embodiment of the invention provides a target area detection method and device, a model training method and device, equipment and a medium. The method comprises the steps that an initial image containing a target area is acquired; determining a target area coordinate corresponding to the initial image and a coordinate weight corresponding to the target area coordinate; obtaining a fused target area coordinate according to a target area coordinate corresponding to the initial image and a coordinate weight corresponding to the target area coordinate; intercepting the initial image according to the coordinates of the fused target area to obtain a target image; target area key point coordinates corresponding to the target image and coordinate weights corresponding to the target area key point coordinates are determined; and according to the target area key point coordinate corresponding to the target image and the coordinate weight corresponding to the target area key point coordinate, obtaining a fused target area key point coordinate. According to the embodiment of the invention, by fusing the data of the infrared image and the visible light image, the advantages of the two images in different environments can be comprehensively utilized, and the accuracy of target area detection is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of face detection technology, and in particular, to a target area detection method, a model training method for target area detection, a target area detection device, a model training device for target area detection, an electronic device, and a computer-readable storage medium. Background Art

[0002] In existing cockpit perception systems, face detection models and face key point detection models usually rely on a single type of image data, such as image data obtained by infrared sensors and image data obtained by visible light sensors.

[0003] However, by comparing the image data obtained by the infrared sensor with the image data obtained by the visible light sensor, it can be seen that the amount of information in the image data obtained by the visible light sensor is greater than that in the image data obtained by the infrared sensor, and the image imaging effect of the infrared sensor at night is better than that of the image data obtained by the visible light sensor. Summary of the invention

[0004] The embodiments of the present invention provide a target area detection method, a model training method, an apparatus, an electronic device and a computer-readable storage medium to solve the problem that existing face detection usually only relies on a single type of image data.

[0005] An embodiment of the present invention discloses a target area detection method, the method comprising:

[0006] Acquire an initial image containing a target area; the initial image includes an initial infrared image and an initial visible light image;

[0007] Determine the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates;

[0008] Obtaining fused target area coordinates according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates;

[0009] According to the coordinates of the fusion target area, the target image is intercepted on the initial image to obtain the target image; the target image includes a target infrared image and a target visible light image;

[0010] Determine the coordinates of the key points of the target area corresponding to the target image and the coordinate weights corresponding to the key points of the target area;

[0011] The fused target area key point coordinates are obtained according to the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates.

[0012] Optionally, determining the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates includes:

[0013] Inputting the initial infrared image into an infrared target area detection model to obtain the infrared target area coordinates corresponding to the initial infrared image;

[0014] Inputting the initial visible light image into a visible light target area detection model to obtain the visible light target area coordinates corresponding to the initial visible light image;

[0015] Determine the light data;

[0016] If the light data exceeds the light threshold, the coordinate weight corresponding to the visible light target area coordinate is set to the first weight, and the coordinate weight corresponding to the infrared target area coordinate is set to the second weight;

[0017] If the light data does not exceed the light threshold, the coordinate weight corresponding to the infrared target area coordinates is set to the first weight, and the coordinate weight corresponding to the visible light target area coordinates is set to the second weight.

[0018] Optionally, the determining the coordinates of the target area key points corresponding to the target image and the coordinate weights corresponding to the target area key points coordinates includes:

[0019] Inputting the target infrared image into the infrared target area key point detection model to obtain the infrared target area key point coordinates corresponding to the target infrared image;

[0020] Inputting the target visible light image into a visible light target region key point detection model to obtain the coordinates of the visible light target region key points corresponding to the target visible light image;

[0021] If the light data exceeds the light threshold, the coordinate weight corresponding to the coordinates of the key point in the visible light target area is set to the first weight, and the coordinate weight corresponding to the coordinates of the key point in the infrared target area is set to the second weight;

[0022] If the light data does not exceed the light threshold, the coordinate weight corresponding to the coordinates of the key point of the infrared target area is set to the first weight, and the coordinate weight corresponding to the coordinates of the key point of the visible light target area is set to the second weight.

[0023] Optionally, obtaining the fused target area coordinates according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates includes:

[0024] Multiplying the infrared target area coordinates and the coordinate weights corresponding to the infrared target area coordinates to obtain a first coordinate value;

[0025] Multiplying the visible light target area coordinates and the coordinate weights corresponding to the visible light target area coordinates to obtain a second coordinate value;

[0026] The first coordinate value and the second coordinate value are added and fused to obtain the fusion target area coordinates.

[0027] Optionally, the first weight is greater than the second weight, and the sum of the first weight and the second weight is a preset value.

[0028] The embodiment of the present invention further discloses a model training method for target area detection, the method comprising:

[0029] Acquire a single-channel infrared image; the single-channel infrared image includes an infrared pixel matrix, and the infrared pixel matrix includes a plurality of infrared pixel points;

[0030] Dividing the infrared pixel matrix into a plurality of target matrices;

[0031] Selecting at least one infrared pixel point from each of the plurality of target matrices as a target infrared pixel point;

[0032] Combining a plurality of target infrared pixels into a target single-channel infrared image;

[0033] Training an infrared target area detection model and an infrared target area key point detection model according to the target single-channel infrared image;

[0034] Acquire a multi-channel visible light image; the multi-channel visible light image includes a plurality of channel visible light images, the channel visible light image includes a visible light pixel matrix, and the visible light pixel matrix includes a plurality of visible light pixel points;

[0035] Dividing the visible light pixel matrix into a plurality of target visible light matrices;

[0036] Selecting at least one visible light pixel point from each of the plurality of target visible light matrices as a target visible light pixel point;

[0037] Combining a plurality of the target visible light pixel points into a target channel visible light image;

[0038] Combining a plurality of the target channel visible light images to obtain a target multi-channel visible light image;

[0039] Training a visible light target area detection model and a visible light target area key point detection model according to the target multi-channel visible light image;

[0040] The infrared target area detection model and the visible light target area detection model are used to obtain target area coordinates, and the infrared target area key point detection model and the visible light target area key point detection model are used to obtain target area key point coordinates.

[0041] The embodiment of the present invention further discloses a target area detection device, the device comprising:

[0042] An initial image module, used to obtain an initial image containing a target area; the initial image includes an initial infrared image and an initial visible light image;

[0043] A target area coordinate module, used to determine the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates;

[0044] A fused target area coordinate module, used to obtain fused target area coordinates according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates;

[0045] A target image module, used to intercept the initial image according to the coordinates of the fusion target area to obtain a target image; the target image includes a target infrared image and a target visible light image;

[0046] A target area key point coordinate module, used to determine the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates;

[0047] The fused target area key point coordinate module is used to obtain the fused target area key point coordinates according to the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates.

[0048] The embodiment of the present invention further discloses a model training device for target area detection, the device comprising:

[0049] A single-channel infrared image acquisition module is used to acquire a single-channel infrared image; the single-channel infrared image includes an infrared pixel matrix, and the infrared pixel matrix includes a plurality of infrared pixel points;

[0050] A target matrix division module, used for dividing the infrared pixel matrix into a plurality of target matrices;

[0051] A target infrared pixel point selection module, used to select at least one infrared pixel point in each of the plurality of target matrices as a target infrared pixel point;

[0052] A target single-channel infrared image module, used for combining a plurality of target infrared pixels into a target single-channel infrared image;

[0053] A first training module is used to train an infrared target area detection model and an infrared target area key point detection model according to the target single-channel infrared image;

[0054] A multi-channel visible light image acquisition module, used to acquire a multi-channel visible light image; the multi-channel visible light image includes a plurality of channel visible light images, the channel visible light image includes a visible light pixel matrix, and the visible light pixel matrix includes a plurality of visible light pixel points;

[0055] A target visible light matrix division matrix module, used for dividing the visible light pixel matrix into a plurality of target visible light matrices;

[0056] A target visible light pixel point selection module, used to select at least one visible light pixel point from each of the plurality of target visible light matrices as a target visible light pixel point;

[0057] A target channel visible light image module, used for combining a plurality of the target visible light pixels into a target channel visible light image;

[0058] A target multi-channel visible light image module, used for combining a plurality of the target channel visible light images to obtain a target multi-channel visible light image;

[0059] A second training module is used to train a visible light target area detection model and a visible light target area key point detection model according to the target multi-channel visible light image;

[0060] The infrared target area detection model and the visible light target area detection model are used to obtain target area coordinates, and the infrared target area key point detection model and the visible light target area key point detection model are used to obtain target area key point coordinates.

[0061] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0062] The memory is used to store computer programs;

[0063] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0064] The embodiment of the present invention further discloses a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the embodiment of the present invention.

[0065] The embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the method described in the embodiment of the present invention.

[0066] The embodiments of the present invention include the following advantages:

[0067] In an embodiment of the present invention, a target area detection method is provided, which includes obtaining an initial image containing a target area; the initial image includes an initial infrared image and an initial visible light image; determining the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates; obtaining fused target area coordinates according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates; intercepting on the initial image according to the fused target area coordinates to obtain a target image; the target image includes a target infrared image and a target visible light image; determining the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates; obtaining fused target area key point coordinates according to the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates. The embodiment of the present invention can more accurately determine the position of the target area by fusing the data of the infrared image and the visible light image, and by introducing the coordinate weights, the advantages of the two images in different environments can be comprehensively utilized to improve the accuracy of target area detection, and image interception can be performed according to the fused target area coordinates, so as to more effectively focus on the target area and detect the key points in the target area, thereby better providing accurate target areas for other perception models, thereby improving the performance and user experience of the entire cockpit perception system.

[0068] In an embodiment of the present invention, a model training method for target area detection is also provided, which comprises acquiring a single-channel infrared image; the single-channel infrared image comprises an infrared pixel matrix, the infrared pixel matrix comprises a plurality of infrared pixel points; dividing the infrared pixel matrix into a plurality of target matrices; selecting at least one infrared pixel point as a target infrared pixel point in each of the plurality of target matrices; composing a target single-channel infrared image with the plurality of target infrared pixel points; training an infrared target area detection model and an infrared target area key point detection model according to the target single-channel infrared image; acquiring a multi-channel visible light image; the multi-channel visible light image comprises a plurality of channel visible light images, the channel visible light image comprises a visible light pixel matrix, the visible light pixel matrix comprises The invention discloses a method for detecting a target area of ​​an infrared target region and a visible light target region by using a plurality of target visible light matrices; dividing the visible light pixel matrix into a plurality of target visible light matrices; selecting at least one visible light pixel in each of the plurality of target visible light matrices as a target visible light pixel; forming a target channel visible light image with the plurality of target visible light pixels; combining the plurality of target channel visible light images to obtain a target multi-channel visible light image; training a visible light target area detection model and a visible light target area key point detection model according to the target multi-channel visible light image; wherein the infrared target area detection model and the visible light target area detection model are used to obtain the target area coordinates, and the infrared target area key point detection model and the visible light target area key point detection model are used to obtain the target area key point coordinates. The embodiments of the present invention can achieve data expansion for an image by performing a series of operations such as matrix division, pixel selection, and pixel reorganization on a single-channel infrared image or a multi-channel visible light image, thereby increasing data diversity, and being able to learn more features with a minimum data set, thereby greatly improving the generalization capabilities of the target area detection model (infrared target area detection model and visible light target area detection model) and the target area key point detection model (infrared target area key point detection model and visible light target area key point detection model). BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flowchart of a target area detection method provided in an embodiment of the present invention;

[0070] Figure 2 It is a flow chart of target area detection based on a mixture of infrared sensors and visible light sensors provided in an embodiment of the present invention;

[0071] Figure 3 is a flowchart of the steps of a model training method for target area detection provided in an embodiment of the present invention;

[0072] Figure 4is a schematic diagram of data expansion based on an infrared pixel point provided in an embodiment of the present invention;

[0073] Figure 5 is a schematic diagram of data expansion based on two infrared pixels provided in an embodiment of the present invention;

[0074] Figure 6 is a schematic diagram of data expansion based on three infrared pixels provided in an embodiment of the present invention;

[0075] Figure 7 is a schematic diagram of data expansion based on a visible light pixel point provided in an embodiment of the present invention;

[0076] Figure 8 is a schematic diagram of data expansion based on two visible light pixels provided in an embodiment of the present invention;

[0077] Fig. 9 is a schematic diagram of data expansion based on three visible light pixels provided in an embodiment of the present invention;

[0078] Fig.10 is a structural block diagram of a target area detection device provided in an embodiment of the present invention;

[0079] Fig.11 is a structural block diagram of a model training device for target area detection provided in an embodiment of the present invention;

[0080] Fig.12 is a schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention;

[0081] Fig.13 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0082] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0083] In the development of modern smart cockpit technology, improving the safety and comfort of drivers and passengers has become a core goal. To achieve this goal, the smart cockpit system needs to be able to accurately identify and track the status of drivers and passengers, especially the detection of faces and the location of key points. As a key information source for individual recognition and emotion analysis, faces play a pivotal role in smart cockpits.

[0084] Therefore, a target area detection method provided in an embodiment of the present invention can be applied to the intelligent perception system of the smart cockpit. By detecting and obtaining the target area coordinates and the target area key point coordinates, it can better provide an accurate ROI area (target area) for other perception models of the cockpit, thereby improving the accuracy and user experience of the entire cockpit perception system. It can be understood that the "target area" mentioned in the embodiment of the present invention can be a human face in the smart cockpit, or it can be other living bodies, such as animals, and the embodiment of the present invention is not limited.

[0085] Reference Figure 1 , shows a flowchart of a target area detection method provided in an embodiment of the present invention, which may specifically include the following steps:

[0086] Step 101, obtaining an initial image including a target area; the initial image includes an initial infrared image and an initial visible light image;

[0087] In practical applications, infrared sensors can detect long-wave infrared radiation that is invisible to the human eye. This radiation still exists in low-light or no-light environments, especially the infrared radiation emitted by the human body. Therefore, infrared images can provide clear facial contours and details at night or in dim light environments, which is essential to ensure that the driver monitoring system can work properly under various lighting conditions. Visible light sensors capture light within the spectrum range visible to the human eye. Therefore, under normal lighting conditions, visible light images can provide rich color information and details. This information is very important for identifying facial features and helps to improve the accuracy of face detection.

[0088] Therefore, in an embodiment of the present invention, in order to provide consistent and accurate target area detection and analysis in various environments, an infrared sensor and a visible light sensor may be provided in the cockpit directly opposite the driver to obtain different types of images, and these two different types of images may be subsequently fused to improve the accuracy of target area detection.

[0089] While the vehicle is driving, the system will obtain images containing the target area (driver's face) from the infrared sensor and the visible light sensor, namely, the initial infrared image and the initial visible light image, respectively.

[0090] Step 102, determining the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates;

[0091] In the embodiment of the present invention, the system uses a pre-trained target region detection model to analyze the initial infrared image and the initial visible light image, and determines the target region coordinates (i.e., the position of the face) in each image. At the same time, according to the current lighting conditions (day or night), the system assigns different weights to the data of each sensor to optimize the detection results. For example, during the day, the weight of the visible light image is greater than the weight of the infrared image, while at night, the weight of the infrared image is greater than the weight of the visible light image.

[0092] The target area coordinates usually refer to the coordinates of the bounding box of the target area (face) detected in the image. The target area coordinates define the position and size of the target area in the image, and are usually represented by the pixel coordinates of the upper left corner and the lower right corner. The target area coordinates are based on the detection results of the entire target area and are mainly used to determine the approximate position of the target area in the image.

[0093] Step 103, obtaining fused target area coordinates according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates;

[0094] In the embodiment of the present invention, the system performs weighted averaging of the target region coordinates of the infrared image and the visible light image according to their respective weights to obtain a fused target region coordinate (i.e., fused target region coordinate), which represents an estimate of the optimal target region position and size under the current lighting conditions. The fused target region coordinate comprehensively considers the data of the two sensors, improving the accuracy and robustness of target region detection, especially when the lighting conditions change.

[0095] The embodiment of the present invention performs coordinate fusion through weighted averaging to obtain fused target area coordinates. The fused target area coordinates integrate the data of the two sensors, reduce the errors that may exist in a single sensor, and thus improve the accuracy of target area detection. Moreover, under different lighting conditions, the system can automatically adjust the weights to ensure that stable target area detection results can be provided in various environments.

[0096] Step 104: intercepting the initial image according to the coordinates of the fusion target area to obtain a target image; the target image includes a target infrared image and a target visible light image;

[0097] In the previous step 103, the system has fused the target area coordinates of the infrared image and the visible light image by weighted averaging to obtain the fused target area coordinates. Since the fused target area coordinates integrate the data of the infrared sensor and the visible light sensor and provide the optimal estimate of the target area position and size under the current lighting conditions, the system will subsequently directly use the fused target area coordinates to determine the target areas in the initial infrared image and the initial visible light image, respectively, which can be understood as a ROI area (Region of Interest).

[0098] When applied to face detection in smart cockpits, ROI can refer to the part of the image that contains the face. This area is usually a rectangular box whose boundaries are defined by the upper left and lower right corners of the fused face coordinates. Specifically, the system uses the fused target area coordinates to define an ROI area, which contains the face part in the image. The system then extracts the pixel data in the ROI area from the initial infrared image and the initial visible light image, respectively, to generate the ROI of the infrared image and the ROI of the visible light image, i.e., the target infrared image and the target visible light image in the embodiment of the present invention, for subsequent key point detection and other analysis.

[0099] In an embodiment of the present invention, the fused target region coordinates integrate the data of the infrared and visible light sensors, and provide an estimate of the optimal target region position and size under the current lighting conditions, which helps to improve the accuracy of target region detection. Secondly, by using the fused target region coordinates to define the ROI area, the system can automatically adjust under different lighting conditions to ensure that stable face detection results can be provided in various environments, thereby enhancing the robustness of the system. In addition, focusing on the target region area to capture the image reduces the interference of background noise, optimizes resource utilization, and improves processing efficiency.

[0100] Step 105, determining the coordinates of the key points of the target area corresponding to the target image and the coordinate weights corresponding to the key points of the target area;

[0101] In an embodiment of the present invention, the system first inputs the captured target infrared image and target visible light image into a pre-trained target area key point detection model respectively. The target area key point detection model analyzes the target area features in the target image and outputs the key point coordinates of the target area in each image, which are expressed in the form of pixel coordinates.

[0102] When applied to face detection in smart cockpits, the coordinates of the key points in the target area provide more detailed geometric structure information of the face, which is crucial for advanced applications such as expression recognition and head posture estimation. The key points in the target area usually include the positions of the eyes, nose, mouth, eyebrows and other parts.

[0103] There is a close connection between the target area coordinates and the target area key point coordinates in the embodiment of the present invention. First, the target area coordinates provide an initial search area for the target area key point detection, ensuring that the target area key point detection model can focus on processing the image area (target image) containing the target area, thereby improving the detection efficiency and accuracy. Secondly, the target area key point coordinates can be used to further optimize the target area coordinates, for example, by adjusting the size and position of the bounding box through the distribution of key points, so that it fits the target area more closely.

[0104] After determining the coordinates of the target area key points in the target infrared image and the target visible light image, the system assigns different weights to the data of each sensor according to the current lighting conditions (such as day or night). For example, during the day, the target area key point coordinates of the visible light image can be given a higher weight because the visible light image provides richer colors and details under normal lighting conditions; while at night, the target area key point coordinates of the infrared image can be given a higher weight because the infrared image performs better under low light conditions.

[0105] By assigning weights to the target area key point coordinates of different sensors, the system can comprehensively utilize the advantages of infrared and visible light sensors under different lighting conditions to improve the accuracy and robustness of key point detection. This weight allocation strategy ensures that the system can provide consistent and reliable key point detection results in various environments, thereby optimizing the performance of the entire cockpit perception system.

[0106] Step 106: Obtain the fused target area key point coordinates according to the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates.

[0107] As described in the aforementioned step 103, in an embodiment of the present invention, the system performs weighted averaging on the target area key point coordinates of the target infrared image and the target visible light image according to their respective weights to obtain a fused target area key point coordinate (i.e., fused target area key point coordinate), which represents the optimal feature position estimate of the target area under the current lighting conditions.

[0108] When applied to face detection in smart cockpits, the target area key point coordinates can be used for a variety of advanced applications, such as expression recognition, head posture estimation, fatigue detection, etc. By using the fused target area key point coordinates, the system can provide consistent and reliable analysis results under different lighting conditions, thereby optimizing the performance of the entire cockpit perception system.

[0109] In the embodiment of the present invention, whether it is the fusion of the target area coordinates or the fusion of the target area key point coordinates, both are fused by weighted averaging, which is simpler and more effective than other fusion methods. The weighted average can dynamically adjust the weight according to the change of lighting conditions, so that the system can provide the best target area detection results both during the day and at night. In addition, the weighted average calculation is simple, easy to implement, and has high calculation efficiency, which is suitable for real-time applications.

[0110] In an embodiment of the present invention, the system outputs the fused target area coordinates and the fused target area key point coordinates as the final target area detection result. When applied to face detection in a smart cockpit, the fused target area coordinates and the fused target area key point coordinates can be used in subsequent cockpit perception systems, such as driver monitoring, fatigue detection and other applications, to provide more accurate target areas for other perception models in the cockpit system. Through the target area detection method provided in an embodiment of the present invention, the system can provide consistent and accurate target area detection and target area key point positioning under different lighting conditions, thereby improving the performance and user experience of the entire cockpit perception system.

[0111] Reference Figure 2 , shows a target area detection flow chart based on a mixture of infrared sensors and visible light sensors provided in an embodiment of the present invention. Specifically, the system obtains an initial image (initial infrared image and initial visible light image) through an infrared sensor and a visible light sensor respectively, and then inputs the initial image into a target area detection model (an infrared target area detection model and a visible light target area detection model) to obtain target area coordinates (infrared target area coordinates and visible light target area coordinates). The infrared target area coordinates and the visible light target area coordinates are fused to obtain fused target area coordinates. Based on the fused target area coordinates, a target image (target infrared image and target visible light image) can be intercepted in the initial image. Then, the target image is input into a target area key point detection model (an infrared target area key point detection model and a visible light target area key point detection model) to obtain target area key point coordinates (infrared target area key point coordinates and visible light target area key point coordinates). The infrared target area key point coordinates and the visible light target area key point coordinates are fused to obtain fused target area key point coordinates.

[0112] In an embodiment of the present invention, an initial image including a target area is obtained; the initial image includes an initial infrared image and an initial visible light image; the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates are determined; the fused target area coordinates are obtained according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates; the target image is intercepted on the initial image according to the fused target area coordinates to obtain the target image; the target image includes a target infrared image and a target visible light image; the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates are determined; the fused target area key point coordinates are obtained according to the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates. The embodiment of the present invention can more accurately determine the position of the target area by fusing the data of the infrared image and the visible light image, and by introducing the coordinate weights, the advantages of the two images in different environments can be comprehensively utilized to improve the accuracy of target area detection, and image interception can be performed according to the fused target area coordinates to more effectively focus on the target area and detect the key points in the target area, thereby better providing accurate target areas for other perception models, thereby improving the performance and user experience of the entire cockpit perception system.

[0113] In one embodiment of the present invention, the step 102 of determining the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates includes:

[0114] Inputting the initial infrared image into an infrared target area detection model to obtain the infrared target area coordinates corresponding to the initial infrared image;

[0115] Inputting the initial visible light image into a visible light target area detection model to obtain the visible light target area coordinates corresponding to the initial visible light image;

[0116] Determine the light data;

[0117] If the light data exceeds the light threshold, the coordinate weight corresponding to the visible light target area coordinate is set to the first weight, and the coordinate weight corresponding to the infrared target area coordinate is set to the second weight;

[0118] If the light data does not exceed the light threshold, the coordinate weight corresponding to the infrared target area coordinates is set to the first weight, and the coordinate weight corresponding to the visible light target area coordinates is set to the second weight.

[0119] In the embodiment of the present invention, for different types of data, the target region detection model includes an infrared target region detection model and a visible light target region detection model.

[0120] The system first inputs the initial infrared image into the pre-trained infrared target area detection model. This model will analyze the initial infrared image and output the position of the target area in the image, that is, the infrared target area coordinates; similarly, the system inputs the initial visible light image into the visible light target area detection model. This model will analyze the initial visible light image and output the position of the target area in the image, that is, the visible light target area coordinates. Next, the system will evaluate the light intensity according to the lighting conditions of the current environment, and then determine the light data of the current environment. According to the comparison between the light data and the preset light threshold, the system will set the weights of the infrared target area coordinates and the visible light target area coordinates. Specifically, if the light data exceeds the light threshold, it means that the current lighting conditions are good (such as daytime). At this time, the system will set the weight of the visible light target area coordinates to the first weight and the weight of the infrared target area coordinates to the second weight. If the light data does not exceed the light threshold, it means that the current lighting conditions are poor (such as night). At this time, the system will set the weight of the infrared target area coordinates to the first weight and the weight of the visible light target area coordinates to the second weight.

[0121] In the process of determining light data, the light intensity of the environment can be directly measured by the built-in light sensor. When applied to face detection in a smart cockpit, the light sensor is usually located at the front or top of the vehicle and can sense the surrounding light level. The system reads the output values ​​of these sensors and uses them as light data.

[0122] In the absence of a dedicated light sensor, the system can estimate light intensity by analyzing infrared or visible light images. For example, the system can calculate the average brightness or contrast of the image to infer the current lighting conditions. This method relies on image processing algorithms to extract useful information from image data. The system can also estimate light intensity in combination with time and location data. For example, the system can infer approximate lighting conditions based on the current time (such as day or night) and the vehicle's geographical location (such as latitude and longitude). Although this method is not as accurate as directly measuring light intensity, it can be used as a reference in some cases.

[0123] In practical applications, the system may combine multiple methods to determine light data. For example, the system may use light sensors and image analysis at the same time, or combine time and location data to improve the accuracy of light data. In the embodiments of the present invention, there is no limitation on the method of determining light data, and light data may be determined by a single method or a combination of different methods according to actual conditions.

[0124] The main purpose of determining the light data in the embodiment of the present invention is to adjust the weights of the infrared and visible light target area coordinates according to the current lighting conditions. In this way, the system can automatically adjust under different lighting conditions to ensure stable and accurate target area detection results during both daytime and nighttime.

[0125] In one embodiment of the present invention, the step 105 of determining the coordinates of the target area key points corresponding to the target image and the coordinate weights corresponding to the coordinates of the target area key points includes:

[0126] Inputting the target infrared image into the infrared target area key point detection model to obtain the infrared target area key point coordinates corresponding to the target infrared image;

[0127] Inputting the target visible light image into a visible light target region key point detection model to obtain the coordinates of the visible light target region key points corresponding to the target visible light image;

[0128] If the light data exceeds the light threshold, the coordinate weight corresponding to the coordinates of the key point in the visible light target area is set to the first weight, and the coordinate weight corresponding to the coordinates of the key point in the infrared target area is set to the second weight;

[0129] If the light data does not exceed the light threshold, the coordinate weight corresponding to the coordinates of the key point of the infrared target area is set to the first weight, and the coordinate weight corresponding to the coordinates of the key point of the visible light target area is set to the second weight.

[0130] In an embodiment of the present invention, the process of determining the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates is similar to the process of determining the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates. Both are performed by analyzing infrared and visible light images and setting weights based on the comparison of light data with light thresholds to adapt to different lighting conditions, which will not be elaborated here.

[0131] In an embodiment of the present invention, the first weight is greater than the second weight, and the sum of the first weight and the second weight is a preset value.

[0132] In the actual fusion process, the sum of the weights must be 1 to ensure that the result of the weighted average is valid. If the sum of the weights is not equal to 1, the fused coordinates may deviate from the actual face position. By ensuring that the sum of the weights is 1, the system can ensure that the fused coordinates are based on a reasonable combination of the two sensor data. Therefore, in an embodiment of the present invention, the preset value is usually 1, which can ensure that the sum of the weights is 1 for weighted average calculation. The coordinate weights corresponding to the coordinates of the target area and the coordinate weights corresponding to the coordinates of the key points of the target area need to be determined based on the actual light data, that is, the first weight is actually determined based on the light data, and the second weight is determined based on the first weight.

[0133] For example, in a daytime scene, because the visible light sensor can capture richer colors and details under normal lighting conditions, it is necessary to set the weight of the visible light target area key point coordinates to be greater than the infrared target area key point coordinates, so as to provide more accurate target area key point detection. At this time, the weight of the visible light target area coordinates can be 0.9, 0.8, 0.7, 0.6 (first weight), then the corresponding weight of the infrared target area coordinates can be 0.1, 0.2, 0.3, 0.4 (second weight). For another example, in a night scene, because the infrared sensor can provide a clear target area outline in a low light or no light environment, it is necessary to set the weight of the infrared target area key point coordinates to be greater than the visible light target area key point coordinates, so as to provide more accurate target area key point detection. At this time, the weight of the infrared target area coordinates can be 0.9, 0.8, 0.7, 0.6 (first weight), then the corresponding weight of the visible light target area coordinates can be 0.1, 0.2, 0.3, 0.4 (second weight).

[0134] In one embodiment of the present invention, the light data includes the light intensity and uniformity of the current environment. Therefore, it is possible to determine whether the current lighting conditions are good from the light data, thereby further determining the numerical value of the first weight. For example, in a daytime scene, if the current lighting conditions are very good, the system may set the weight of the key point coordinates of the visible light target area to 0.9, and the weight of the key point coordinates of the infrared target area to 0.1; if the lighting conditions are slightly worse, the system may set the weight of the key point coordinates of the visible light target area to 0.6, and the weight of the key point coordinates of the infrared target area to 0.4. For another example, in a night scene, if the current night lighting conditions are very poor, the system may set the weight of the key point coordinates of the infrared target area to 0.9, and the weight of the key point coordinates of the visible light target area to 0.1; if the night lighting conditions are slightly better, the system may set the weight of the key point coordinates of the infrared target area to 0.6, and the weight of the key point coordinates of the visible light target area to 0.4.

[0135] When allocating weights in the embodiment of the present invention, the specific values ​​of the coordinates of the key points of the infrared target area (such as 0.1 or 0.4) depend on the current lighting conditions. The sum of the weights must be 1 to ensure that the result of the weighted average is valid. By dynamically adjusting the weights according to the lighting conditions, the system can provide the best target area key point detection results in different environments.

[0136] In one embodiment of the present invention, the step 103 of obtaining the fused target area coordinates according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates includes:

[0137] Multiplying the infrared target area coordinates and the coordinate weights corresponding to the infrared target area coordinates to obtain a first coordinate value;

[0138] Multiplying the visible light target area coordinates and the coordinate weights corresponding to the visible light target area coordinates to obtain a second coordinate value;

[0139] The first coordinate value and the second coordinate value are added and fused to obtain the fusion target area coordinates.

[0140] In an embodiment of the present invention, first, the infrared target area coordinates are multiplied by their corresponding coordinate weights to adjust the infrared target area coordinates according to the importance of the infrared image under the current lighting conditions (represented by the coordinate weights), and a weighted coordinate value is obtained, referred to as a first coordinate value. Next, the visible light target area coordinates are multiplied by their corresponding coordinate weights to adjust the visible light target area coordinates according to the importance of the visible light image under the current lighting conditions, and another weighted coordinate value is obtained, referred to as a second coordinate value. Finally, the first coordinate value and the second coordinate value are added, that is, the weighted coordinate values ​​of the infrared image and the visible light image are combined to obtain a fused target area coordinate that comprehensively considers the data of the two sensors, and this fused coordinate represents the optimal target area position estimate under the current lighting conditions.

[0141] Assuming that the coordinates of the visible light target area are (x1, y1), the coordinates of the infrared target area are (x2, y2), and the coordinate weight corresponding to the coordinates of the visible light target area is m1, then the coordinate weight corresponding to the coordinates of the infrared target area is (1-m1). At this time, the fused target area coordinates (x, y) are obtained by fusing the following formulas (1) and (2):

[0142] x=x1*m1+x2*(1-m1) (1)

[0143] y=y1*m1+y2*(1-m1) (2)

[0144] In this way, the embodiment of the present invention can dynamically adjust the weights of infrared and visible light target area coordinates according to the current lighting conditions, thereby providing the best target area detection results in different environments. This weight setting strategy ensures that the system can provide stable and accurate target area detection during the day and at night, optimizing the performance of the entire cockpit perception system.

[0145] In another embodiment of the present invention, the step 106 of obtaining the fused target area key point coordinates according to the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates includes:

[0146] Multiplying the infrared target area key point coordinates and the coordinate weights corresponding to the infrared target area key point coordinates to obtain a third coordinate value;

[0147] Multiplying the coordinates of the key point in the visible light target area and the coordinate weight corresponding to the coordinates of the key point in the visible light target area to obtain a fourth coordinate value;

[0148] The third coordinate value and the fourth coordinate value are added and fused to obtain the key point coordinates of the fusion target area.

[0149] In the embodiment of the present invention, the process of obtaining the coordinates of the key points of the fused target area is similar to the process of obtaining the coordinates of the fused target area, and both of them perform data fusion in a weighted average manner to obtain the optimal target area detection result under different lighting conditions.

[0150] Assuming that the coordinates of the key points in the visible light target area are (x3, y3), the coordinates of the key points in the infrared target area are (x4, y4), and the coordinate weight corresponding to the coordinates of the key points in the visible light target area is m2, then the coordinate weight corresponding to the coordinates of the key points in the infrared target area is (1-m2). At this time, the coordinates of the target area in the fused image (x', y') are obtained by fusing the following formulas (3) and (4):

[0151] x'=x3*m2+x4*(1-m2) (3)

[0152] y'=y3*m2+y4*(1-m2) (4)

[0153] In this way, the embodiment of the present invention can dynamically adjust the weights of the infrared and visible light target area key point coordinates according to the current lighting conditions, thereby providing the best target area key point detection results in different environments, ensuring that the system can provide stable and accurate target area key point detection both during the day and at night, and optimizing the performance of the entire cockpit perception system.

[0154] In an embodiment of the present invention, an initial image including a target area is obtained; the initial image includes an initial infrared image and an initial visible light image; the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates are determined; the fused target area coordinates are obtained according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates; the target image is intercepted on the initial image according to the fused target area coordinates to obtain the target image; the target image includes a target infrared image and a target visible light image; the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates are determined; the fused target area key point coordinates are obtained according to the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates. The embodiment of the present invention can more accurately determine the position of the target area by fusing the data of the infrared image and the visible light image, and by introducing the coordinate weights, the advantages of the two images in different environments can be comprehensively utilized to improve the accuracy of target area detection, and image interception can be performed according to the fused target area coordinates to more effectively focus on the target area and detect the key points in the target area, thereby better providing accurate target areas for other perception models, thereby improving the performance and user experience of the entire cockpit perception system.

[0155] Reference Figure 3 , shows a flowchart of a model training method for target area detection provided in an embodiment of the present invention, which may specifically include the following steps:

[0156] Step 301, acquiring a single-channel infrared image; the single-channel infrared image includes an infrared pixel matrix, and the infrared pixel matrix includes a plurality of infrared pixel points;

[0157] Step 302: dividing the infrared pixel matrix into a plurality of target matrices;

[0158] Step 303, selecting at least one infrared pixel point from each of the plurality of target matrices as a target infrared pixel point;

[0159] Step 304, combining the plurality of target infrared pixels into a target single-channel infrared image;

[0160] Step 305: training an infrared target area detection model and an infrared target area key point detection model according to the target single-channel infrared image;

[0161] Step 306: Acquire a multi-channel visible light image; the multi-channel visible light image includes a plurality of channel visible light images, the channel visible light image includes a visible light pixel matrix, and the visible light pixel matrix includes a plurality of visible light pixel points;

[0162] Step 307: Divide the visible light pixel matrix into a plurality of target visible light matrices;

[0163] Step 308: Select at least one visible light pixel point from each of the plurality of target visible light matrices as a target visible light pixel point;

[0164] Step 309, composing a plurality of the target visible light pixel points into a target channel visible light image;

[0165] Step 3010: combining a plurality of the target channel visible light images to obtain a target multi-channel visible light image;

[0166] Step 3011: training a visible light target area detection model and a visible light target area key point detection model according to the target multi-channel visible light image;

[0167] The infrared target area detection model and the visible light target area detection model are used to obtain target area coordinates, and the infrared target area key point detection model and the visible light target area key point detection model are used to obtain target area key point coordinates.

[0168] In practical applications, infrared images usually have only one channel, because infrared sensors detect infrared radiation, which is in a wavelength range that is invisible to the human eye. The single-channel nature of infrared images means that each pixel only records a value, which represents the intensity of the infrared radiation received by the pixel. In some cases, infrared images may be represented as three channels (for example, R, G, B channels), but the pixel values ​​in each channel are the same. The reason for this is to enable infrared images to be displayed and processed in standard image processing software, because most image processing software and display devices are designed for color images and expect images to have three channels. Copying a single-channel infrared image into three channels can create a grayscale image in which the pixel values ​​in each channel are the same, so that the image looks gray but can be displayed correctly on a color display. This representation method does not increase the amount of information in the image, but is only for compatibility and visualization purposes.

[0169] Therefore, the pixel value of each pixel in the three channels of the infrared image is the same.

[0170] In an embodiment of the present invention, before realizing target area detection, it is necessary to pre-train the infrared target area detection model and the infrared target area key point detection model. Specifically, an infrared image will be selected and data enhancement will be performed to expand the data training set. In this way, more features can be learned using the smallest data set, thereby greatly improving the generalization ability of the infrared target area detection model and the infrared target area key point detection model.

[0171] Since the pixel value of each pixel in the three channels of the infrared image is the same, when performing data enhancement on the infrared image, only the single-channel infrared image is enhanced. The specific process of data enhancement is as follows:

[0172] Assume that a single-channel infrared image contains a 6*6 infrared pixel matrix, and each value in the infrared pixel matrix represents the pixel value of each channel.

[0173] Reference Figure 4 , shows a schematic diagram of data expansion based on an infrared pixel point provided in an embodiment of the present invention. Specifically, in the infrared pixel matrix 401, an infrared pixel point 403 is randomly selected from the upper left, lower left, upper right and lower right 3*3 target matrices 402 to form a new channel image. Then, the single-channel infrared image will be expanded. A single-channel infrared image 404 of a target.

[0174] Reference Figure 5 , shows a schematic diagram of data expansion based on two infrared pixels provided in an embodiment of the present invention. Specifically, in the infrared pixel matrix 501, two infrared pixels 503 are randomly selected from the upper left, lower left, upper right and lower right 3*3 target matrix 502 to form a new channel image. Here, pixels can be selected in the horizontal direction, vertical direction, diagonal direction and anti-diagonal direction. Then, the single-channel infrared image will be expanded. A single-channel infrared image 504 of a target.

[0175] Reference Figure 6 , shows a schematic diagram of data expansion based on three infrared pixels provided in an embodiment of the present invention. Specifically, in the infrared pixel matrix 601, three infrared pixels 603 are randomly selected from the upper left, lower left, upper right and lower right 3*3 target matrix 602 to form a new channel image. Here, pixel points can also be selected in the horizontal direction, vertical direction, diagonal direction and anti-diagonal direction. Then, the single-channel infrared image will be expanded. A single-channel infrared image 604 of a target.

[0176] In an embodiment of the present invention, a single-channel infrared image may be data expanded in the above manner, including randomly selecting n pixels (where n=1-9) in a 3x3 matrix to form a new channel image (target single-channel infrared image).

[0177] Therefore, a single-channel infrared image can be simply and effectively expanded in the above way, which increases the data diversity and enables more features to be learned with the smallest data set, so that the generalization ability of the infrared target area detection model and the infrared target area key point detection model can be greatly improved.

[0178] In practical applications, visible light images are different from infrared images. Visible light images are RGB (Red, Green, Blue) images. In RGB color images, each pixel consists of three color channels: red (R), green (G), and blue (B). These three channels represent the three primary colors of light, and their combination can produce various colors. The pixel value of each channel represents the intensity of the color channel, and the value range is usually 0 to 255, where 0 means there is no light of that color and 255 means the light intensity of that color is the highest. Since the RGB value of each pixel can change independently, the color of each pixel can be different, that is, in the same image, different pixels can have different colors because their RGB value combinations are different.

[0179] Therefore, the values ​​of each point in the R channel map, G channel map, and B channel map obtained after channel separation of the RGB image are also different.

[0180] In an embodiment of the present invention, before realizing target area detection, it is necessary to pre-train the visible light target area detection model and the visible light target area key point detection model. Specifically, a visible light image will be selected and data enhancement will be performed to achieve the expansion of the data training set. In this way, more features can be learned using the smallest data set, thereby greatly improving the generalization ability of the visible light target area detection model and the visible light target area key point detection model.

[0181] Since the pixel values ​​of each pixel in the three channels of the visible light image are different, when performing data enhancement on the visible light image, it is necessary to perform data enhancement on each channel image (channel visible light image) in the visible light image. The specific process of data enhancement is as follows:

[0182] Assume that the channel visible light image contains a visible light pixel matrix of size 6*6, and each value in the visible light pixel matrix represents a pixel value of each channel.

[0183] Reference Figure 7 , shows a schematic diagram of data expansion based on a visible light pixel point provided in an embodiment of the present invention. Specifically, in a visible light pixel matrix 701, a visible light pixel point 703 is randomly selected from the upper left, lower left, upper right and lower right 3*3 target visible light matrices 702 to form a new channel image. Then, each channel visible light image will be expanded A target channel visible light image 704 is obtained, and then the target channel visible light images on each channel are combined to obtain a target multi-channel visible light image.

[0184] Reference Figure 8 , shows a schematic diagram of data expansion based on two visible light pixel points provided in an embodiment of the present invention. Specifically, in a visible light pixel matrix 801, two visible light pixel points 803 are randomly selected from the upper left, lower left, upper right and lower right 3*3 target visible light matrix 802 to form a new channel image. Here, pixel points can be selected in the horizontal direction, vertical direction, diagonal direction and anti-diagonal direction. Then, each channel visible light image will be expanded. A target channel visible light image 804 is obtained, and then the target channel visible light images on each channel are combined to obtain a target multi-channel visible light image.

[0185] Reference Fig. 9 , shows a schematic diagram of data expansion based on three visible light pixel points provided in an embodiment of the present invention. Specifically, in a visible light pixel matrix 901, three visible light pixel points 903 are randomly selected from the upper left, lower left, upper right and lower right 3*3 target visible light matrix 902 to form a new channel image. Here, pixel points can also be selected in the horizontal direction, vertical direction, diagonal direction and anti-diagonal direction. Then, each channel visible light image will be expanded. A target channel visible light image 904 is obtained, and then the target channel visible light images on each channel are combined to obtain a target multi-channel visible light image.

[0186] In an embodiment of the present invention, a multi-channel visible light image may be data expanded in the above manner, including randomly selecting n pixels (where n=1-9) in a 3x3 matrix to form a new channel image (target multi-channel visible light image).

[0187] Therefore, a multi-channel visible light image can be simply and effectively expanded in the above-mentioned way, which increases the data diversity and enables more features to be learned with the smallest data set, thereby greatly improving the generalization ability of the visible light target area detection model and the visible light target area key point detection model.

[0188] In one embodiment of the present invention, the target region detection model and the target region key point detection model need to be iteratively trained. The specific iterative process is as follows:

[0189] First, 50,000 positive samples and 50,000 negative samples are selected as the initial training set, and the initial model is trained with these data.

[0190] Secondly, the initial model is used to screen the 1 million data set: when screening positive samples, positive samples with scores below 0.1 are added to the initial training set for training; when screening negative samples, negative samples with scores greater than 0.9 are added to the initial training set for training. The number of positive and negative samples added each time is 3,000. Before adding to the initial training set, the positive and negative samples to be added will be checked to ensure that there are no negative samples mixed with positive samples, and no positive samples mixed with negative samples.

[0191] Then, when the threshold of positive samples is set to 0.1 and the threshold of negative samples is set to 0.9, if the number of positive and negative samples screened is less than 1000, the threshold of positive samples is adjusted to 0.2 and the threshold of negative samples is adjusted to 0.8. This process will be repeated, using thresholds of 0.1, 0.2, 0.3, and 0.4 to screen positive samples, and thresholds of 0.9, 0.8, 0.7, and 0.6 to screen negative samples. Before adding to the initial training set, the positive and negative samples to be added will also be checked to ensure the purity of the data set.

[0192] Finally, the model is used to detect false positive videos on the platform, and undetected positive and negative samples (i.e., false positive and missed samples) are added to the streamlined data set for training, and the detected positive and negative samples are added to the 1 million data set for subsequent iterations. When adding undetected positive and negative samples to the initial training set, the added data is carefully checked to ensure that there are no negative samples mixed with positive samples, and no positive samples mixed with negative samples.

[0193] In the embodiment of the present invention, a "positive sample" refers to sample data that includes the target area, while a "negative sample" refers to sample data that does not include the target area. Through the above iteration method, the model can continuously learn from new data to improve its accuracy and robustness in detecting the target area.

[0194] In an embodiment of the present invention, a single-channel infrared image is acquired; the single-channel infrared image includes an infrared pixel matrix, and the infrared pixel matrix includes a plurality of infrared pixel points; the infrared pixel matrix is ​​divided into a plurality of target matrices; at least one infrared pixel point is selected from each of the plurality of target matrices as a target infrared pixel point; the plurality of target infrared pixel points are combined into a target single-channel infrared image; an infrared target area detection model and an infrared target area key point detection model are trained according to the target single-channel infrared image; a multi-channel visible light image is acquired; the multi-channel visible light image includes a plurality of channel visible light images, and the channel visible light image includes a visible light pixel matrix, and the visible light pixel matrix includes a plurality of visible light pixel points; The visible light pixel matrix is ​​divided into a plurality of target visible light matrices; at least one visible light pixel point is selected from each of the plurality of target visible light matrices as a target visible light pixel point; the plurality of target visible light pixels are combined into a target channel visible light image; the plurality of target channel visible light images are combined to obtain a target multi-channel visible light image; a visible light target area detection model and a visible light target area key point detection model are trained according to the target multi-channel visible light image; wherein the infrared target area detection model and the visible light target area detection model are used to obtain target area coordinates, and the infrared target area key point detection model and the visible light target area key point detection model are used to obtain target area key point coordinates. The embodiments of the present invention can achieve data expansion for an image by performing a series of operations such as matrix division, pixel selection, and pixel reorganization on a single-channel infrared image or a multi-channel visible light image, thereby increasing data diversity, and being able to learn more features with a minimum data set, thereby greatly improving the generalization capabilities of the target area detection model (infrared target area detection model and visible light target area detection model) and the target area key point detection model (infrared target area key point detection model and visible light target area key point detection model).

[0195] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0196] Reference Fig.10 , shows a structural block diagram of a target area detection device provided in an embodiment of the present invention, which may specifically include the following modules:

[0197] An initial image module 1001 is used to obtain an initial image containing a target area; the initial image includes an initial infrared image and an initial visible light image;

[0198] A target region coordinate module 1002 is used to determine the target region coordinates corresponding to the initial image and the coordinate weights corresponding to the target region coordinates;

[0199] A fused target region coordinate module 1003 is used to obtain fused target region coordinates according to the target region coordinates corresponding to the initial image and the coordinate weights corresponding to the target region coordinates;

[0200] A target image module 1004 is used to intercept the initial image according to the coordinates of the fused target area to obtain a target image; the target image includes a target infrared image and a target visible light image;

[0201] A target area key point coordinate module 1005 is used to determine the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates;

[0202] The fused target region key point coordinates module 1006 is used to obtain fused target region key point coordinates according to the target region key point coordinates corresponding to the target image and the coordinate weights corresponding to the target region key point coordinates.

[0203] Reference Fig.11 , shows a structural block diagram of a model training device for target area detection provided in an embodiment of the present invention, which may specifically include the following modules:

[0204] The single-channel infrared image acquisition module 1101 is used to acquire a single-channel infrared image; the single-channel infrared image includes an infrared pixel matrix, and the infrared pixel matrix includes a plurality of infrared pixel points;

[0205] A target matrix division module 1102 is used to divide the infrared pixel matrix into a plurality of target matrices;

[0206] A target infrared pixel point selection module 1103 is used to select at least one infrared pixel point from each of the plurality of target matrices as a target infrared pixel point;

[0207] A target single-channel infrared image module 1104 is used to combine a plurality of target infrared pixels into a target single-channel infrared image;

[0208] The first training module 1105 is used to train an infrared target area detection model and an infrared target area key point detection model according to the target single-channel infrared image;

[0209] A multi-channel visible light image acquisition module 1106 is used to acquire a multi-channel visible light image; the multi-channel visible light image includes a plurality of channel visible light images, the channel visible light image includes a visible light pixel matrix, and the visible light pixel matrix includes a plurality of visible light pixel points;

[0210] A target visible light matrix division matrix module 1107 is used to divide the visible light pixel matrix into a plurality of target visible light matrices;

[0211] A target visible light pixel point selection module 1108 is used to select at least one visible light pixel point from each of the plurality of target visible light matrices as a target visible light pixel point;

[0212] A target channel visible light image module 1109 is used to combine a plurality of the target visible light pixels into a target channel visible light image;

[0213] A target multi-channel visible light image module 11010 is used to combine a plurality of the target channel visible light images to obtain a target multi-channel visible light image;

[0214] The second training module 11011 is used to train a visible light target area detection model and a visible light target area key point detection model according to the target multi-channel visible light image;

[0215] The infrared target area detection model and the visible light target area detection model are used to obtain target area coordinates, and the infrared target area key point detection model and the visible light target area key point detection model are used to obtain target area key point coordinates.

[0216] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0217] In addition, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement the various processes of the above-mentioned face detection method embodiment when executing the program stored in the memory, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0218] The embodiment of the present invention further provides a computer-readable storage medium, on which instructions are stored, and when executed by one or more processors, the processors execute the various processes of the above-mentioned face detection method embodiment, and can achieve the same technical effect, to avoid repetition, no further description is given here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0219] An embodiment of the present invention also provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned face detection method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0220] Fig.12 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0221] The electronic device 1200 includes but is not limited to: a radio frequency unit 1201, a network module 1202, an audio output unit 1203, an input unit 1204, a sensor 1205, a display unit 1206, a user input unit 1207, an interface unit 1208, a memory 1209, a processor 1210, and a power supply 1211. Those skilled in the art will appreciate that Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In the embodiments of the present invention, the electronic device includes but is not limited to a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted terminal, a wearable device, and a pedometer.

[0222] It should be understood that in the embodiment of the present invention, the radio frequency unit 1201 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving downlink data from the base station, it is sent to the processor 1210 for processing; in addition, uplink data is sent to the base station. Generally, the radio frequency unit 1201 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 1201 can also communicate with the network and other devices through a wireless communication system.

[0223] The electronic device provides users with wireless broadband Internet access through the network module 1202, such as helping users to send and receive emails, browse web pages, and access streaming media.

[0224] The audio output unit 1203 can convert the audio data received by the RF unit 1201 or the network module 1202 or stored in the memory 1209 into an audio signal and output it as sound. Moreover, the audio output unit 1203 can also provide audio output related to a specific function performed by the electronic device 1200 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 1203 includes a speaker, a buzzer, a receiver, etc.

[0225] The input unit 1204 is used to receive audio or video signals. The input unit 1204 may include a graphics processor (GPU) 12041 and a microphone 12042. The graphics processor 12041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 1206. The image frame processed by the graphics processor 12041 can be stored in the memory 1209 (or other storage medium) or sent via the radio frequency unit 1201 or the network module 1202. The microphone 12042 can receive sound and can process such sound into audio data. The processed audio data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 1201 in the case of a telephone call mode.

[0226] The electronic device 1200 also includes at least one sensor 1205, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 12061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 12061 and / or the backlight when the electronic device 1200 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used to identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; the sensor 1205 can also include a fingerprint sensor, a pressure sensor, an iris sensor, a molecular sensor, a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0227] The display unit 1206 is used to display information input by the user or information provided to the user. The display unit 1206 may include a display panel 12061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0228] The user input unit 1207 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the user input unit 1207 includes a touch panel 12071 and other input devices 12072. The touch panel 12071, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch panel 12071). The touch panel 12071 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact point coordinates, and then sends it to the processor 1210, receives the command sent by the processor 1210 and executes it. In addition, the touch panel 12071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 12071, the user input unit 1207 may also include other input devices 12072. Specifically, other input devices 12072 may include but are not limited to physical keyboards, function keys (such as volume control keys, switch keys, etc.), trackballs, mice, and joysticks, which will not be repeated here.

[0229] Furthermore, the touch panel 12071 may be overlaid on the display panel 12061. When the touch panel 12071 detects a touch operation on or near it, it is transmitted to the processor 1210 to determine the type of touch event. Then, the processor 1210 provides corresponding visual output on the display panel 12061 according to the type of touch event. Fig.12 In the figure, the touch panel 12071 and the display panel 12061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 12071 and the display panel 12061 can be integrated to realize the input and output functions of the electronic device, which is not limited here.

[0230] The interface unit 1208 is an interface for connecting an external device to the electronic device 1200. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 1208 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the electronic device 1200 or may be used to transmit data between the electronic device 1200 and an external device.

[0231] The memory 1209 can be used to store software programs and various data. The memory 1209 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1209 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0232] The processor 1210 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 1209 and calling data stored in the memory 1209, it performs various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. The processor 1210 may include one or more processing units; preferably, the processor 1210 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1210.

[0233] The electronic device 1200 may also include a power supply 1211 (such as a battery) for supplying power to each component. Preferably, the power supply 1211 may be logically connected to the processor 1210 via a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system.

[0234] In addition, the electronic device 1200 includes some functional modules not shown, which will not be described in detail here.

[0235] like Fig.13As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1301 is also provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the face detection method described in the above embodiment.

[0236] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0237] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0238] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

[0239] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the embodiments of the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0240] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0241] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0242] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0243] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0244] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0245] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A target area detection method, characterized in that: The method comprises: Acquire an initial image containing a target area; the initial image includes an initial infrared image and an initial visible light image; Determine the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates; Obtaining fused target area coordinates according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates; According to the coordinates of the fusion target area, the target image is intercepted on the initial image to obtain the target image; the target image includes a target infrared image and a target visible light image; Determine the coordinates of the key points of the target area corresponding to the target image and the coordinate weights corresponding to the key points of the target area; The fused target area key point coordinates are obtained according to the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates.

2. The method according to claim 1, characterized in that The determining the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates includes: Inputting the initial infrared image into an infrared target area detection model to obtain the infrared target area coordinates corresponding to the initial infrared image; Inputting the initial visible light image into a visible light target area detection model to obtain the visible light target area coordinates corresponding to the initial visible light image; Determine the light data; If the light data exceeds the light threshold, the coordinate weight corresponding to the visible light target area coordinate is set to the first weight, and the coordinate weight corresponding to the infrared target area coordinate is set to the second weight; If the light data does not exceed the light threshold, the coordinate weight corresponding to the infrared target area coordinates is set to the first weight, and the coordinate weight corresponding to the visible light target area coordinates is set to the second weight.

3. The method according to claim 2, characterized in that The determining the coordinates of the target area key points corresponding to the target image and the coordinate weights corresponding to the target area key points coordinates includes: Inputting the target infrared image into the infrared target area key point detection model to obtain the infrared target area key point coordinates corresponding to the target infrared image; Inputting the target visible light image into a visible light target region key point detection model to obtain the coordinates of the visible light target region key points corresponding to the target visible light image; If the light data exceeds the light threshold, the coordinate weight corresponding to the coordinates of the key point in the visible light target area is set to the first weight, and the coordinate weight corresponding to the coordinates of the key point in the infrared target area is set to the second weight; If the light data does not exceed the light threshold, the coordinate weight corresponding to the coordinates of the key point of the infrared target area is set to the first weight, and the coordinate weight corresponding to the coordinates of the key point of the visible light target area is set to the second weight.

4. The method according to claim 3, characterized in that The step of obtaining the fused target area coordinates according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates includes: Multiplying the infrared target area coordinates and the coordinate weights corresponding to the infrared target area coordinates to obtain a first coordinate value; Multiplying the visible light target area coordinates and the coordinate weights corresponding to the visible light target area coordinates to obtain a second coordinate value; The first coordinate value and the second coordinate value are added and fused to obtain the fusion target area coordinates.

5. The method according to claim 3, characterized in that: The first weight is greater than the second weight, and the sum of the first weight and the second weight is a preset value.

6. A model training method for target area detection, characterized in that: The method comprises: Acquire a single-channel infrared image; the single-channel infrared image includes an infrared pixel matrix, and the infrared pixel matrix includes a plurality of infrared pixel points; Dividing the infrared pixel matrix into a plurality of target matrices; Selecting at least one infrared pixel point from each of the plurality of target matrices as a target infrared pixel point; Combining a plurality of target infrared pixels into a target single-channel infrared image; Training an infrared target area detection model and an infrared target area key point detection model according to the target single-channel infrared image; Acquire a multi-channel visible light image; the multi-channel visible light image includes a plurality of channel visible light images, the channel visible light image includes a visible light pixel matrix, and the visible light pixel matrix includes a plurality of visible light pixel points; Dividing the visible light pixel matrix into a plurality of target visible light matrices; Selecting at least one visible light pixel point from each of the plurality of target visible light matrices as a target visible light pixel point; Combining a plurality of the target visible light pixel points into a target channel visible light image; Combining a plurality of the target channel visible light images to obtain a target multi-channel visible light image; Training a visible light target area detection model and a visible light target area key point detection model according to the target multi-channel visible light image; The infrared target area detection model and the visible light target area detection model are used to obtain target area coordinates, and the infrared target area key point detection model and the visible light target area key point detection model are used to obtain target area key point coordinates.

7. A target area detection device, characterized in that: The device comprises: An initial image module, used to obtain an initial image containing a target area; the initial image includes an initial infrared image and an initial visible light image; A target area coordinate module, used to determine the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates; A fused target area coordinate module, used to obtain fused target area coordinates according to the target area coordinates corresponding to the initial image and the coordinate weights corresponding to the target area coordinates; A target image module, used to intercept the initial image according to the coordinates of the fusion target area to obtain a target image; the target image includes a target infrared image and a target visible light image; A target area key point coordinate module, used to determine the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates; The fused target area key point coordinate module is used to obtain the fused target area key point coordinates according to the target area key point coordinates corresponding to the target image and the coordinate weights corresponding to the target area key point coordinates.

8. A model training device for target area detection, characterized in that: The device comprises: A single-channel infrared image acquisition module is used to acquire a single-channel infrared image; the single-channel infrared image includes an infrared pixel matrix, and the infrared pixel matrix includes a plurality of infrared pixel points; A target matrix division module, used for dividing the infrared pixel matrix into a plurality of target matrices; A target infrared pixel point selection module, used to select at least one infrared pixel point in each of the plurality of target matrices as a target infrared pixel point; A target single-channel infrared image module, used for combining a plurality of target infrared pixels into a target single-channel infrared image; A first training module is used to train an infrared target area detection model and an infrared target area key point detection model according to the target single-channel infrared image; A multi-channel visible light image acquisition module, used to acquire a multi-channel visible light image; the multi-channel visible light image includes a plurality of channel visible light images, the channel visible light image includes a visible light pixel matrix, and the visible light pixel matrix includes a plurality of visible light pixel points; A target visible light matrix division matrix module, used for dividing the visible light pixel matrix into a plurality of target visible light matrices; A target visible light pixel point selection module, used to select at least one visible light pixel point from each of the plurality of target visible light matrices as a target visible light pixel point; A target channel visible light image module, used for combining a plurality of the target visible light pixels into a target channel visible light image; A target multi-channel visible light image module, used for combining a plurality of the target channel visible light images to obtain a target multi-channel visible light image; A second training module is used to train a visible light target area detection model and a visible light target area key point detection model according to the target multi-channel visible light image; The infrared target area detection model and the visible light target area detection model are used to obtain target area coordinates, and the infrared target area key point detection model and the visible light target area key point detection model are used to obtain target area key point coordinates.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor, when used to execute a program stored in the memory, implements the method according to any one of claims 1 to 5, or the method according to claim 6.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to execute the method according to any one of claims 1 to 5, or the method according to claim 6.

Images