Face capture and monitoring method, system and device based on dual spectrum

Abstract

The present invention provides a method and system, device and storage medium for face capture and monitoring based on dual spectrum, including: obtaining image data of infrared spectrum and visible spectrum; fusing the image data to obtain optimal image data; generating alarm information according to the optimal image data, the alarm information at least including the rectangular area coordinates of the face location; cutting the optimal image data according to the rectangular area coordinates to generate a face picture. The dual spectrum camera has the same effect as an ordinary camera when there is sufficient light, but can still take color pictures when there is insufficient light or no light, and can also effectively reduce light pollution and improve the quality of face images at night. When capturing on the road, there is no need to compensate for strong light, which causes pedestrians or vehicles to be dizzy and dazzled, thereby increasing the customer experience and improving satisfaction.

Description

Technical Field

[0001] The present invention belongs to a security system and relates to a monitoring method and equipment, and in particular to a dual-spectrum-based face capture and monitoring method and system, equipment and storage medium. Background Art

[0002] Traditional color cameras only collect and use information in the visible light band. In low-light, hazy and other scenes, as the visible light intensity decreases or the scene information is blocked, the color image effect will be seriously degraded; in these scenes, through proper exposure control, clear infrared images with a high signal-to-noise ratio can be obtained. However, on the other hand, infrared images only have grayscale information but no color information. In addition, traditional face capture cameras are affected by the illumination at night (insufficient light or no light), which affects the effect of capturing face images. In order to shoot at night, many ordinary lenses will add infrared lights for fill light, resulting in black and white images, or use stronger flash lights for fill light to obtain color images. Summary of the invention

[0003] The technical problem to be solved by the present invention is: in view of the problems existing in the prior art, the present invention provides a method for improving the quality of nighttime face images.

[0004] In a first aspect, an embodiment of the present application provides a dual-spectrum-based face capture and monitoring method, the method comprising:

[0005] Acquire image data of infrared spectrum and visible spectrum;

[0006] fusing the image data to obtain optimal image data;

[0007] Generate alarm information according to the optimal image data, wherein the alarm information at least includes coordinates of a rectangular area where the face is located;

[0008] The optimal image data is cropped according to the rectangular area coordinates to generate a face image.

[0009] In a second aspect, the embodiment of the present application further provides a dual-spectrum face capture and monitoring system based on dual-spectrum, the system comprising:

[0010] Acquisition module: acquires image data of infrared spectrum and visible spectrum;

[0011] Fusion module: fusing the image data to obtain optimal image data;

[0012] Generating module: generating alarm information according to the optimal image data, wherein the alarm information at least includes the coordinates of the rectangular area where the face is located;

[0013] Cropping module: Cropping the optimal image data according to the rectangular area coordinates to generate a face image.

[0014] In the third aspect, an embodiment of the present application also provides a dual-spectrum based face capture and monitoring device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements each step of the dual-spectrum based face capture and monitoring method as described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, each step in the dual-spectrum-based face capture and monitoring method as described in the first aspect is implemented.

[0016] The embodiment of the present application provides a dual-spectrum-based face capture and monitoring method, which includes: obtaining image data of infrared spectrum and visible spectrum; fusing the image data to obtain optimal image data; generating alarm information based on the optimal image data, the alarm information at least including the rectangular area coordinates of the face location; and cropping the optimal image data according to the rectangular area coordinates to generate a face picture. The dual-spectrum camera has the same effect as an ordinary camera when there is sufficient light, but can still capture color images when there is insufficient light or no light. It can also effectively reduce light pollution and improve the quality of face images at night. When capturing on the road, there is no need to compensate for strong light, which causes pedestrians or vehicles to be dizzy or dazzled, thereby increasing the customer experience and improving satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The specific structure of the present invention is described in detail below with reference to the accompanying drawings

[0018] Figure 1 It is a flow chart of the dual-spectrum-based face capture and monitoring method of the present invention;

[0019] Figure 2 It is a schematic diagram of a sub-process of the dual-spectrum-based face capturing and monitoring method of the present invention;

[0020] Figure 3 It is another sub-process diagram of the dual-spectrum-based face capture and monitoring method of the present invention;

[0021] Figure 4 It is another sub-process diagram of the dual-spectrum-based face capture and monitoring method of the present invention;

[0022] Figure 5 It is another sub-process diagram of the dual-spectrum-based face capture and monitoring method of the present invention;

[0023] Figure 6 The figure is a schematic diagram of the program modules of the dual-spectrum-based face capturing and monitoring method of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, features and advantages of the invention more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] Figure 1 : is a flow chart of a dual-spectrum-based face capture and monitoring method in an embodiment of the present application. In this embodiment, the dual-spectrum-based face capture and monitoring method includes:

[0026] Step 101: Acquire image data of infrared spectrum and visible spectrum.

[0027] Among them, in this embodiment, a dual-spectrum camera is used to collect images from two image sensors, one of which is visible light and the other is infrared light, and the images collected by the two image sensors are processed to generate bright YUV format color images. The image sensor collects image data in a method in which the light signal passes through the lens and then passes through a dual-light prism, and is split and beam-split on the inclined surface of the dual-light prism, that is, the infrared light passes through the prism and reaches the CMOS image sensor that receives the infrared light, and the image brightness signal is obtained. After the visible light is reflected on the inclined surface of the prism, it reaches the CMOS image sensor that receives the visible light, and the image color signal is obtained. The YUV is a color encoding method, which is often used in various video processing components. When YUV encodes photos or videos, it takes into account the human perception ability and allows the bandwidth of chromaticity to be reduced.

[0028] Step 102: Fusing the image data to obtain optimal image data.

[0029] In this embodiment, in a certain scene, the image signal of visible light and the image signal of infrared light are fused to obtain the optimal image data, but it is not limited to a certain scene. In fact, the scene classification calculation is first performed, such as ordinary scenes, visible light scenes at night, scenes with infrared reflective objects at night, and ordinary scenes at night, and then the optimal scene image data is synthesized to output the video frame data at that moment.

[0030] Step 103: Generate alarm information based on the optimal image data, wherein the alarm information at least includes the coordinates of the rectangular area where the face is located.

[0031] The generating of alarm information specifically includes that the original video collected by the image sensor is input to the face detection unit in the YUV format data frame, wherein YUV is a color encoding method, which is often used in various video processing components. When encoding photos or videos, YUV allows the bandwidth of chrominance to be reduced in consideration of human perception. The face detection unit is responsible for analyzing each frame of YUV raw data generated by the camera and outputting alarm information, which includes not only the coordinates of the rectangular area where the face is located.

[0032] Step 104: crop the optimal image data according to the rectangular region coordinates to generate a face image.

[0033] Among them, cropping and generating a face image within the rectangular area coordinates specifically includes cropping the specific position of the face from the rectangular coordinate position in the original video in the YUV format at the same time as the face detection is performed, that is, cropping the face image in the YUV format from the complete video frame image, and then using the MJPEG encoder to encode the face image in the YUV format into image data in the JPG format.

[0034] The embodiment of the present application provides a dual-spectrum-based face capture and monitoring method, which includes: obtaining image data of infrared spectrum and visible spectrum; fusing the image data to obtain optimal image data; generating alarm information based on the optimal image data, the alarm information at least including the rectangular area coordinates of the face location; and cropping the optimal image data according to the rectangular area coordinates to generate a face picture. The dual-spectrum camera has the same effect as an ordinary camera when there is sufficient light, but can still capture color images when there is insufficient light or no light. It can also effectively reduce light pollution and improve the quality of face images at night. When capturing on the road, there is no need to compensate for strong light, which causes pedestrians or vehicles to be dizzy or dazzled, thereby increasing the customer experience and improving satisfaction.

[0035] Further, based on the above embodiment, referring to Figure 2 , Figure 2 : is a schematic diagram of a sub-process of a dual-spectrum face capture and monitoring method in this embodiment. In this embodiment, the specific steps of obtaining optimal image data include:

[0036] The light passes through the lens and then through the bifocal prism, where it is split and beam-split on the inclined surface of the bifocal prism;

[0037] The image sensor obtains an image color signal of the visible spectrum and an image brightness signal of the infrared spectrum;

[0038] Performing color processing on the image color signal of the visible spectrum and the image brightness signal of the infrared spectrum respectively, and processing the image data in MIPI format to obtain image data in RAW format;

[0039] Then, white balance processing is performed on the image color signal of the visible spectrum and the image brightness signal of the infrared spectrum after color processing, and the image data in RAW format is processed to obtain image data in RGB format;

[0040] Performing histogram statistics on the image color signal of the visible spectrum and the image brightness signal of the infrared spectrum after white balance processing to obtain the color distribution of the visible spectrum and infrared spectrum image data, and processing the image data in RGB format to obtain image data in YUV format;

[0041] The image color signal of the visible spectrum and the image brightness signal of the infrared spectrum after white balance processing are fused with the image data after color distribution of the image color signal of the visible spectrum and the image brightness signal of the infrared spectrum to obtain the optimal image data in YUV format.

[0042] Further, based on the above embodiment, referring to Figure 3 , Figure 3 The figure is a sub-flow diagram of the face capture and monitoring method based on dual spectrum in this embodiment. In this embodiment, the above step 103 generates alarm information according to the optimal image data, and the alarm information at least includes the coordinates of the rectangular area where the face is located. The original video captured by the camera is in YUV format, and the original video in YUV format is input to the face detection unit as a data frame. The face detection unit is responsible for analyzing each frame of YUV raw data generated by the camera, and then outputting the alarm information, and caching the alarm information to the event list.

[0043] Specifically, the alarm information includes not only the coordinates of the rectangular area where the face is located, but also a unique number and a timestamp, wherein the unique number is a unique number for each person, and the timestamp is the time at that time.

[0044] Further, based on the above embodiment, referring to Figure 4 , Figure 4 : is a schematic diagram of a sub-process of a dual-spectrum face capture and monitoring method in this embodiment. In this embodiment, it includes the specific steps of detecting whether a face exists, and cropping the optimal image data according to the rectangular area coordinates to generate a face picture:

[0045] Firstly, the original video in YUV format is encoded to generate a video in JPG format, and the video frames and video frame timestamps in JPG format are obtained;

[0046] At the same time, detect whether there is a human face in the original video in the complete YUV format;

[0047] If there is a face, an alarm message is output, wherein the alarm message includes the coordinates of the rectangular area where the face is located, the unique number of the face, and the timestamp;

[0048] Then, the coordinates of the face rectangle area in the alarm information are cropped to obtain the face image and face timestamp in the current YUV format;

[0049] The current face image in YUV format is encoded to obtain a face image in JPG format.

[0050] Further, based on the above embodiment, referring to Figure 5 , Figure 5 This is a sub-flow chart of the dual-spectrum face capture and monitoring method in this embodiment, including specific steps of outputting face images based on 5G transmission mode:

[0051] First, a connection is established between the camera and the backend server. The backend server sends a connection request to the specified camera. After receiving the request, the camera agrees and returns a response.

[0052] The backend server sends a request for video and pictures to the specified camera. After receiving the request, the camera sends the video and pictures to the backend server.

[0053] The backend server sends a disconnect request to the specified camera. After receiving the request, the camera agrees to disconnect and returns a response.

[0054] Furthermore, the embodiment of the present application also provides a dual-spectrum-based face capture and monitoring device 200, referring to Figure 6 , Figure 6 Schematic diagram of a dual-spectrum face capture and monitoring device module in an embodiment of the present application. In this embodiment, the dual-spectrum face capture and monitoring device 200 includes:

[0055] Acquisition module 201: Acquisition of infrared spectrum and visible spectrum image data;

[0056] Fusion module 202: fusing the image data to obtain optimal image data;

[0057] Generating module 203: generating alarm information according to the optimal image data, wherein the alarm information at least includes the coordinates of the rectangular area where the face is located;

[0058] The cropping module 204 crops the optimal image data according to the rectangular region coordinates to generate a face image.

[0059] The dual-spectrum-based face capture and monitoring device 200 provided in the embodiment of the present application can achieve: obtaining image data of infrared spectrum and visible spectrum; fusing the image data to obtain optimal image data; generating alarm information based on the optimal image data, the alarm information at least including the rectangular area coordinates of the face location; cropping the optimal image data according to the rectangular area coordinates to generate a face picture. The dual-spectrum camera has the same effect as an ordinary camera when there is sufficient light, but can still take color pictures when there is insufficient light or no light. It can also effectively reduce light pollution and improve the quality of face images at night. When capturing on the road, there is no need to compensate for strong light, which causes pedestrians or vehicles to be dizzy and dazzled, thereby increasing the customer experience and improving satisfaction.

[0060] Furthermore, an embodiment of the present application also provides a dual-spectrum-based face capture and monitoring device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step of the above-mentioned dual-spectrum-based face capture and monitoring method is implemented.

[0061] Furthermore, the present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the various steps of the above-mentioned dual-spectrum-based face capture and monitoring method are implemented.

[0062] Each functional module in each embodiment of the present invention can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0063] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0064] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0065] The above is a description of a dual-spectrum-based face capture and monitoring method and system, device and storage medium provided by the present invention. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A face capture and monitoring method based on dual spectrum, characterized in that: The method comprises: Acquire image data of infrared spectrum and visible spectrum, wherein image brightness signal of infrared spectrum and image color signal of visible spectrum are acquired; Processing the image brightness signal of the infrared spectrum and the image color signal of the visible spectrum, and then performing color processing on the image signals of the infrared spectrum and the visible spectrum respectively; Performing white balance processing on the two image signals after color image processing respectively; Perform histogram statistics on the two image signals after white balance processing to obtain the color distribution of the two image signals; fusing the image data to obtain optimal image data, wherein the two image signals after white balance processing and the color distribution of the two image signals are fused to obtain the optimal image data; Generate alarm information according to the optimal image data, wherein the alarm information at least includes coordinates of a rectangular area where the face is located; The optimal image data is cropped according to the rectangular area coordinates to generate a face image.

2. The method according to claim 1, characterized in that After generating the alarm information according to the optimal image data, the method further includes: The alarm information is stored.

3. The method according to claim 1, characterized in that The alarm information also includes a unique number and a timestamp.

4. The method according to claim 1, characterized in that After the fused image data is cropped according to the rectangular area coordinates to generate a face image, the method further includes: Output the face image, wherein the outputting the face image is based on a 5G transmission method.

5. A face capture and monitoring system based on dual spectrum, characterized in that: The system comprises: Acquisition module: acquires image data of infrared spectrum and visible spectrum, wherein image brightness signal of infrared spectrum and image color signal of visible spectrum are acquired; Processing the image brightness signal of the infrared spectrum and the image color signal of the visible spectrum, and then performing color processing on the image signals of the infrared spectrum and the visible spectrum respectively; Performing white balance processing on the two image signals after color image processing respectively; Perform histogram statistics on the two image signals after white balance processing to obtain the color distribution of the two image signals; Fusion module: fusing the image data to obtain optimal image data, wherein the two image signals after white balance processing and the color distribution of the two image signals are fused to obtain the optimal image data; Generating module: generating alarm information according to the optimal image data, wherein the alarm information at least includes the coordinates of the rectangular area where the face is located; Cropping module: Cropping the optimal image data according to the rectangular area coordinates to generate a face image.

6. A dual-spectrum-based face capture and monitoring device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, each step of the dual-spectrum-based face capture and monitoring method as described in any one of claims 1 to 4 is implemented.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the dual-spectrum-based face capture and monitoring method as described in any one of claims 1 to 4 is implemented.

Images