A video processing system, a face recognition device and method

Abstract

This invention discloses a video processing system that implements some video and image processing steps using multiple independent dedicated circuits and on-chip memory to reduce CPU consumption, thereby achieving the goals of reducing power consumption and processing latency. It includes a control module, a dedicated processing module, and a target detection and recognition module. The control module sequentially calls specified functional modules according to task requirements. The target detection and recognition module performs target detection and recognition on the pre-processed image. The dedicated processing module includes at least one of the following modules: a first image size scaling module, a second image size scaling module, a color space conversion module, an image flipping module, and a display module. Each dedicated processing module includes an independent circuit to implement a predetermined function, and its input source address and output target address are stored in on-chip memory.

Description

Technical Field

[0001] This invention relates to the field of video and image processing technology, and in particular to a video processing system, a face recognition device and method. Background Technology

[0002] With the development of video processing technology, facial recognition technology has become increasingly mature and is being applied more and more widely in various fields. Compared with other biometric identification methods, facial recognition has the characteristics of being non-invasive, thus becoming the most widely used biometric technology. In recent years, facial recognition technology has played a core role in areas such as security authentication, electronic payment, smart locks, and remote conferencing.

[0003] Facial recognition technology primarily processes video images and videos through machine learning. This machine learning requires significant computing power, leading to substantial power consumption. Currently, common video image and video processing solutions often employ a general-purpose CPU+GPU approach, where the GPU provides deep learning computing power while the CPU handles image pre- and post-processing. This approach results in large power consumption and a large footprint, hindering the miniaturization trend of devices. To reduce power consumption, some lightweight solutions exist, employing an NPU+DSP implementation. This sacrifices some performance and versatility to reduce power consumption, but the DSP used for image pre-processing still exhibits significant power consumption and latency. Summary of the Invention

[0004] To address some or all of the problems in the prior art, the first aspect of this invention provides a video processing system that implements some video and image processing steps by using multiple independent dedicated circuits and on-chip memory, thereby reducing CPU consumption and achieving the goals of reducing power consumption and processing latency. The video processing system includes:

[0005] The control module is used to sequentially call specified functional modules according to task requirements;

[0006] The target detection and recognition module is used to perform target detection and recognition on the preprocessed image; and

[0007] At least one dedicated processing module is provided for preprocessing and / or postprocessing images. Each dedicated processing module includes an independent circuit to perform a predetermined function. The source address of the input and the target address of the output of the circuit are stored in on-chip memory.

[0008] Furthermore, the dedicated processing module includes: a first image size scaling module, a second image size scaling module, a color space conversion module, an image flipping module, and a display module.

[0009] Furthermore, the target detection and recognition module includes at least one neural network structure.

[0010] Furthermore, the first image size scaling module includes a jump point sampling circuit, which is used to read data at equal intervals from the source address of the on-chip memory and store it in the target address to reduce the image size by a specified factor.

[0011] Furthermore, the jump-point sampling circuit reads one byte of data at a time, and the interval between two adjacent data is 2. n , where n takes the value of an integer from 0 to 4.

[0012] Furthermore, the second image size scaling module includes a bilinear interpolation circuit and a column weighted multiplication circuit, and the second image size scaling module is used to adjust the image to a specified size.

[0013] Furthermore, the color space conversion module is used to convert the YUV422 data stream into a BGR888 or RGB444 data stream.

[0014] Furthermore, the image flipping module includes a reverse circuit for flipping the image horizontally.

[0015] Based on the video processing system described above, a second aspect of the present invention provides a face recognition device, which includes an image acquisition device and an image processing chip, wherein the image processing chip includes the video processing system described above to perform face recognition on the image acquired by the image acquisition device.

[0016] A third aspect of the present invention provides a method for performing face recognition using the face recognition device as described above, comprising:

[0017] The image acquisition device acquires video in real time, and the image processing chip captures images from the video at specified intervals, comparing two consecutive frames to perform motion detection.

[0018] If movement is detected, the image processing chip enters activation mode and performs face recognition on the most recently captured image; and

[0019] If there is no movement, the image processing chip remains in a low-power state.

[0020] Furthermore, the motion detection includes:

[0021] The captured image is compared with the image stored in on-chip memory:

[0022] If the difference between two frames exceeds a threshold, then motion is considered to have occurred; and

[0023] If the difference between two frames does not exceed the threshold, it is considered that there is no movement, the image is stored in the on-chip memory, and the previous frame image is overwritten.

[0024] Furthermore, the motion detection also includes:

[0025] The captured image is scaled using the first image scaling module.

[0026] Furthermore, the facial recognition includes:

[0027] The target detection module detects faces in the captured images and outputs the face detection results.

[0028] The control module calculates the confidence score of each face bounding box in the face detection results, selects face bounding boxes with a confidence score greater than a preset value for non-maximum suppression, and then selects the one with the largest area ratio as the input for face recognition; and

[0029] Face recognition is performed using the target recognition module, and the results are displayed.

[0030] Furthermore, the facial recognition also includes:

[0031] Before face detection, the captured image is scaled and its colors are converted using the first image size scaling module and the color space conversion module.

[0032] Furthermore, the face recognition via the target recognition module includes:

[0033] The transformation matrix of the selected face bounding box is calculated using a five-point alignment method;

[0034] The face frame is resized using the second image size scaling module and converted to a specified format using the color space conversion module.

[0035] A symmetrical image is generated using the image flipping module, and then converted to a specified format using the color space conversion module.

[0036] Feature vectors are extracted from the specified format face bounding boxes and their symmetrical images, and then merged; and

[0037] The face recognition result is obtained by querying matching face data from the database based on the merged feature vector.

[0038] Furthermore, the query for matching facial data includes:

[0039] The merged feature vector is multiplied by each set of face data in the database, and the set of face data with the largest result is taken as the recognition result. If the calculated result is less than the preset value, the matching fails.

[0040] This invention provides a video processing system that specifically designs circuits for certain steps in image preprocessing and post-processing, such as scaling, color space conversion, and flipping. It simplifies and abstracts specific algorithms into byte operations, rather than overall function-level operations. Through the combination of these miniature circuits abstracted from specific algorithms, the corresponding functions can be completed while maintaining low power consumption and area. Furthermore, the video processing system employs dedicated on-chip circuitry, operating by directly accessing SRAM (on-chip memory), which effectively reduces transmission latency. Attached Figure Description

[0041] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0042] Figure 1 A schematic diagram of the structure of a video processing system according to an embodiment of the present invention is shown; and

[0043] Figure 2 The diagram shows a flowchart of a face recognition method according to an embodiment of the present invention. Detailed Implementation

[0044] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or in conjunction with other alternatives and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific numbers and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details.

[0045] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0046] It should be noted that the embodiments of the present invention describe the method steps in a specific order; however, this is only for illustrating the specific embodiment and not for limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0047] To reduce CPU consumption and achieve the goals of reducing power consumption and processing latency, this invention provides a video processing system that adopts a new vector processor architecture suitable for video processing. This architecture has dedicated image processing circuits and deep learning post-processing circuits to implement some video processing steps using dedicated processing chips and dedicated memory.

[0048] In this invention, the term "independent" means that a dedicated processing module, when performing its designated function, can complete the function without external input or intervention after the execution process begins, except for receiving initial input. Such functions include, for example, image scaling, color space conversion, image flipping, display, etc.

[0049] The present invention will be further described below with reference to the accompanying drawings of the embodiments.

[0050] Figure 1 A schematic diagram of the structure of a video processing system according to an embodiment of the present invention is shown. Figure 1 As shown, a video processing system includes a control module 101, a target detection and recognition module 102, and a dedicated processing module 103. The video processing system also includes on-chip memory 104. In one embodiment of the present invention, the on-chip memory includes static random-access memory (SRAM), which is used to store image data. The dedicated processing module 103 performs corresponding byte operations based on the image data stored in the on-chip memory 104, thereby realizing the specified processing operation.

[0051] The control module 101, also known as the general processing module, can sequentially call specified functional modules according to task requirements to control the overall video processing process. On the other hand, it can also be internally configured with corresponding structures to implement some simple algorithms required in video processing, such as confidence calculation, sorting, etc. In other words, all processing and / or operations other than those performed by the target detection and recognition module 102 and the dedicated processing module 103 are implemented through the control module 101.

[0052] The target detection and recognition module 102 is mainly used for image target detection and recognition. For example, in face recognition applications, face detection and recognition can be performed through the target detection and recognition module. In one embodiment of the present invention, the target detection and recognition are both implemented through a corresponding neural network structure; therefore, the target detection and recognition module can also be called a neural network processing module.

[0053] The dedicated processing module 103 is used for preprocessing and / or postprocessing the image. Each dedicated processing module includes an independent circuit to implement a predetermined function, such as data dot multiplication, jump sampling, and reversal, thereby realizing image scaling, flipping, and other processing. In an embodiment of the present invention, in order to reduce processing latency, the source address of the circuit input and the target address of the output are both stored in the on-chip memory 104.

[0054] In embodiments of the present invention, the video processing system includes, but is not limited to, one or more of the following dedicated processing modules 103: a first image size scaling module 131, a second image size scaling module 132, a color space conversion module 133, an image flipping module 134, a color combining module 135, and a display module 136.

[0055] In one embodiment of the present invention, the first image size scaling module 131 includes a jump-point sampling circuit, which is used to read data from the source address of the on-chip memory at equal intervals and store it in the target address to reduce the image size by a specified multiple. Specifically, the jump-point sampling circuit takes 1 byte from the source address located in the on-chip memory SRAM at intervals of several bytes and places it in the target address at a specified interval step, for a total of several times. The interval between reading and storing data is 1, 2, 4, 8, or 16 bytes. Except for the moved bytes, the other unmoved bytes retain the original data. Based on this, image size scaling can be achieved. For example, for a YUV422 format image, by setting its input jump point to 4, output jump point to 4, and output jump point to 2 through the jump-point sampling circuit, the Y component can be extracted; by setting the input jump point to 8 and output jump point to 4, the U and V components can be extracted. In this way, by three sampling jump points, a 2x scaling of the YUV422 image can be achieved.

[0056] In one embodiment of the present invention, the second image size scaling module 132 includes a bilinear interpolation circuit and a batch weighted multiplication circuit. The bilinear interpolation circuit outputs the interpolated value to a target address based on specified coordinates and weights of a specified length. The specified coordinates are integers, and the weights are fixed-point decimals. In one embodiment of the present invention, the specified coordinates and weights each have 12 bits for x and y, totaling 24 bits. The batch weighted multiplication circuit is used to perform batch calculations of weighted multiplication. In one embodiment of the present invention, for each multiplication, the input consists of four unsigned 8-bit integers and four 8-bit fixed-point decimals. The corresponding positions of these integers are multiplied and then summed, outputting an unsigned 8-bit integer. Through the cooperation of the bilinear interpolation circuit and the batch weighted multiplication circuit, the image can be adjusted to a specified size.

[0057] The color space conversion module 133 is used to convert the color space of an image. In one embodiment of the present invention, the color space conversion module 133 is used to retrieve a fixed-length YUV422 data stream of pixels from the source address of the on-chip memory, convert it to a BGR888 data stream or an RGB444 data stream through a corresponding conversion circuit, and place it at the target address. The BGR888 data stream is generally used as the input of a neural network, and the RGB444 data stream is generally used as the display output. The specific conversion process is as follows:

[0058] The Y, U, and V values ​​of each pixel in the YUV422 data stream are converted using the following formula:

[0059] R = Y + ((358 (V-128))>>8);

[0060] G = Y - ((88 (U-128) + 182 (V-128))>>8);

[0061] B = Y + ((453 (U-128))>>8);

[0062] After conversion, if the value of R, G, or B is greater than 255 or less than 0, it will be set to 255 or 0 respectively. This converts the YUV422 data stream to a BGR888 data stream. The value of each channel in the RGB444 data stream is obtained by right-shifting the corresponding value in the BGR888 data stream by 4 bits.

[0063] In one embodiment of the present invention, the image flipping module 134 includes a reversal circuit. After reading an RGB image line by line into the reversal circuit, the input address and output address are set to the same address, thus enabling the image to be flipped horizontally in place. In one embodiment of the present invention, the horizontal image flipping can be used in target detection and recognition processes to achieve data enhancement. Specifically, the image flipping includes: creating a symmetrical HWC-order RGB image of the image data, that is, first arranging by column, then by row of each column, then by different channels of each point, then moving the first three RGB bytes of each row of data to the last three bytes, and simultaneously moving the last three bytes to the first three bytes, and so on, until the data reversal is completed, thus achieving the horizontal image flipping.

[0064] In one embodiment of the present invention, the color combining module 135 is used to merge two vectors of the same length. It is mainly used for data enhancement in the target detection and recognition process. Specifically, the color combining module calculates the data in the second address bit by bit according to a fixed rule and puts it into the first address. If the data of a certain bit in the second address is 0, the data of the corresponding bit in the first address is not changed.

[0065] In one embodiment of the present invention, the video processing system may further include a camera and a display, which means that the video processing system needs to allocate dedicated memory to the camera or the display through the display module 136 to store the output results of the camera or the results that need to be output by the display.

[0066] Compared to existing CPU+GPU or NPU+DSP processor architectures, the video processing system in this invention simplifies and abstracts specific algorithms into byte operations, implements these byte operations through microcircuits, and finally completes the function while maintaining low power consumption and area through the combination of these microcircuits. Furthermore, the video processing system employs dedicated on-chip circuitry, operating by directly accessing SRAM (on-chip memory), which effectively reduces transmission latency.

[0067] The video processing system described above can be applied to low-power face detection and recognition chips, and further analyzed for application in face recognition devices such as smart door locks and smart access control systems. Face recognition devices typically include an image acquisition device and an image processing chip. The image processing chip can be equipped with the video processing system described above to perform face recognition on the images acquired by the image acquisition device. To achieve face recognition and subsequently grant permissions based on the recognition results, in one embodiment of the present invention, the ROM of the image processing chip contains a model and facial feature data, which are loaded into the PSRAM after system startup.

[0068] Figure 2 This diagram illustrates a flow chart of a face recognition method according to an embodiment of the present invention. The steps within the dashed boxes are optional and can be omitted as needed in different embodiments of the invention. Figure 2 As shown, for the face recognition device described above, the method for performing face recognition includes:

[0069] First, in step 201, motion detection. To reduce power consumption, the image acquisition device in a face recognition device, such as a camera, is usually kept on. However, its internal face recognition and face detection modules are in a low-power state (Always-On, hereinafter referred to as "AO state") most of the time. Typically, the face detection and face recognition modules are only activated when someone passes in front of the camera, i.e., when movement occurs in front of the camera. Therefore, motion detection is required. In one embodiment of the present invention, motion detection can be implemented by the control module, which captures images from the real-time video acquired by the image acquisition device at specified intervals and compares two consecutive frames to perform motion detection. In one embodiment of the present invention, motion detection includes:

[0070] The captured image is compared with the image stored in the on-chip memory. If the difference between the two frames exceeds a threshold, it is considered that there is movement; otherwise, it is considered that there is no movement, and the image is stored in the on-chip memory to overwrite the previous frame.

[0071] In one embodiment of the present invention, in order to reduce the size of on-chip memory, the captured image is typically scaled by a first image scaling module before motion detection. For example, the camera typically uses a 640p resolution. The image is output in 480 YUV422@30fps mode. Before motion detection, the captured image can be scaled to 320 using the first image size scaling module. Size 240;

[0072] Next, in step 202, face recognition. After entering the activation state, face recognition begins. In one embodiment of the present invention, the face recognition includes:

[0073] First, in step 221, face detection. The target detection module detects faces in the captured image. In one embodiment of the invention, a model is loaded from PSRAM and image data is loaded from SRAM into the neural network structure PIM in the target detection module. Then, face detection is performed in PIM according to predetermined steps, and the face detection result is output. In this embodiment, existing face detection algorithms well-known to those skilled in the art can be used to implement face detection. To facilitate face detection, in one embodiment of the invention, step 220, image preprocessing, is performed before face detection. The captured image is scaled and its colors are converted using a first image size scaling module and a color space conversion module. Specifically, each frame of the image is scaled to 320 using a jump-point sampling circuit. The image is 240 pixels in size, and then converted to a BGR888 image using a color space conversion module to facilitate face detection.

[0074] Next, in step 222, a face bounding box is selected. Since multiple faces may appear in the same image, or the face detection result may output multiple face bounding boxes, and for applications such as smart door locks, the focus is usually on identifying individuals attempting to unlock the door. In other words, in such applications, usually only one face needs to be identified. Based on actual application scenarios, this individual is usually closest to the door lock, meaning their face area is usually the largest. Therefore, in one embodiment of the present invention, the face with the largest area in the face detection result is usually selected as the input for face recognition. Furthermore, since some non-faces may also exist in the face detection result, in one embodiment of the present invention, after obtaining the face detection result, the confidence level of each face bounding box in the face detection result can be calculated by the control module. Face bounding boxes with a confidence level greater than a preset value are selected for non-maximum suppression, and then the one with the largest area is selected as the input for face recognition. The preset value can be directly set to a certain value, or the confidence level value can be set according to the desired number of face bounding boxes.

[0075] Next, in step 223, face recognition. Face recognition is performed through the target recognition module. The target recognition module extracts the feature vectors of the faces within the selected face frame and compares them with the loaded face feature data to identify the face. In one embodiment of the invention, the feature vectors are multiplied by each set of face feature data in the database, and the set of face data with the largest result is taken as the recognition result. If all the calculated results are less than a preset value, the matching fails. To improve the accuracy of face recognition, in one embodiment of the invention, before face recognition, step 224, data augmentation, is performed, i.e., a symmetrical image is generated using an image flipping module. To facilitate face recognition, a series of preprocessing steps are usually performed before the image is input to the target recognition module, including size scaling and color space conversion. In one embodiment of the invention, before recognition, the transformation matrix of the face image within the selected face frame is calculated using a five-point alignment method. Then, the size of the face image is adjusted to 112 using the bilinear interpolation circuit and the dot product circuit in the second image size scaling module. 112, and use the image flipping module to generate a symmetrical image, then use the color space conversion module to convert them to BGR888 format, then perform face recognition on the two face images separately, outputting 128-dimensional vectors, and then merge them into a 256-dimensional vector which is input into the target recognition module; and

[0076] Finally, in step 225, the image is displayed. In some face recognition devices, the recognition results can also be displayed. Specifically, the detected face bounding box is generated into a new layer by the color combination module, moved to the display module, and merged with the base image for output display.

[0077] The above facial recognition devices and methods can be applied to smart locks, smart access control systems, etc. Taking a smart access control system as an example, a smart lock typically includes a camera, a processing chip, and a lock. The camera remains constantly on. To reduce power consumption, the facial recognition and facial detection systems in the processing chip are mostly off, with only the motion detection module remaining on. When someone passes in front of the camera, the motion detection module detects movement and triggers the facial detection and facial recognition modules. The facial detection and facial recognition modules determine if the user is legitimate. If the user is legitimate, an unlock signal is sent to the lock; otherwise, unlocking is refused, and the facial information is recorded. This type of smart access control can be used in general buildings, structures, offices, homes, parks, etc., and can also be used for car door locks, car trunks, motorcycles, electric vehicles, etc., for authentication. Image verification can be used alone, or it can be combined with voice, fingerprint, etc., to increase security.

[0078] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A video processing system, characterized by, The video processing system comprises: a control module configured to sequentially call specified function modules according to task requirements; a dedicated processing module configured to independently pre-process and / or post-process images, wherein source addresses of inputs of the dedicated processing module and target addresses of outputs of the dedicated processing module are stored in on-chip memory of the video processing system, the dedicated processing module comprises a first image size scaling module, a second image size scaling module, a color space conversion module, an image flipping module, and a display module, wherein the first image size scaling module is configured to scale image size, the first image size scaling module comprises a skip-point sampling circuit configured to read data at equal intervals from a source address of the on-chip memory and store the data into a target address to reduce image size by a specified multiple, the second image size scaling module is configured to scale image size, the color space conversion module is configured to convert color space of an image, the image flipping module is configured to flip an image, and the display module is configured to display an image; and a target detection and recognition module configured to detect and recognize targets in a pre-processed image.

2. The video processing system of claim 1, wherein, The target detection and recognition module comprises at least one neural network structure.

3. The video processing system of claim 1, wherein, The skip-point sampling circuit reads one byte of data at a time, and the interval between adjacent two data is 2n, wherein n is an integer from 0 to 4.

4. The video processing system of claim 1, wherein, The second image size scaling module comprises a bilinear interpolation circuit and a column weighting multiplication circuit, and is configured to adjust an image to a specified size.

5. The video processing system of claim 1, wherein, The color space conversion module is configured to convert YUV422 data stream into BGR888 or RGB444 data stream, wherein converting YUV422 data stream into BGR888 data stream comprises: converting Y, U, and V values of each pixel in YUV422 data stream into R, G, and B values according to the following formula: R = Y + ((358 (V-128)) >> 8); G = Y - ((88 (U-128) + 182 (V-128))>>8); and B = Y + ((453 (U-128))>>8); and converting YUV422 data stream into BGR444 data stream comprises: right shifting the converted BGR888 value of each channel by 4 bits.

6. The video processing system of claim 1, wherein, The image flipping module comprises a reverse order circuit configured to flip an image left and right.

7. A face recognition apparatus, characterized by comprising: The image processing chip comprises the video processing system as claimed in any one of claims 1 to 6 to perform face recognition on images acquired by the image acquisition device.

8. A method for operating a face recognition device according to claim 7, characterized in that The method comprises the steps of: acquiring a video in real time by an image acquisition device, and the image processing chip captures images from the video at specified intervals and compares two adjacent images to detect movement: If there is movement, the image processing chip enters an active mode, processes the latest captured image and performs face recognition, wherein the face recognition includes: detecting a face in the captured image by a target detection module, outputting a face detection result, the face detection result including at least one face frame, then calculating the confidence of each face frame in the face detection result by a control module, selecting a face frame with a confidence greater than a preset value for non-maximum suppression, then selecting the one with the largest area ratio as the input of a target recognition module, calculating the conversion matrix of the selected face frame in a five-point alignment manner by the target recognition module, adjusting the size of the face frame by a second image size scaling module, converting it to a specified format by a color space conversion module, generating a symmetric image by an image flipping module, converting it to a specified format by a color space conversion module, extracting feature vectors from the specified format face frame and its symmetric image respectively, merging them, and finally querying matching face data from the database according to the merged feature vectors to obtain a face recognition result and display it. If there is no movement, the image processing chip remains in a low-power state.

9. The method of claim 8, wherein, The movement detection includes: Comparing the captured image with the image stored in the on-chip memory: If the difference between the two images exceeds the threshold, it is considered that there is movement; and If the difference between the two images does not exceed the threshold, it is considered that there is no movement, the image is stored in the on-chip memory, and the previous frame image is overwritten.

10. The method of claim 9, wherein, The movement detection also includes: By a first image size scaling module, the captured image is scaled in size.

11. The method of claim 8, wherein, Also includes: Before face detection, the captured image is scaled in size and color converted by a first image size scaling module and a color space conversion module.

12. The method of claim 8, wherein, Querying matching face data includes: Dot product operation is performed on the merged feature vector and each group of face data in the database, and the group of face data with the maximum calculation result value is taken as the recognition result, if the calculation result is less than the preset value, it means that the matching fails.

Images