Method for Portrait Tracking and Locking and Portrait Zooming of Conventional Fixed-Point and Fixed-Focus Cameras

Abstract

The present invention discloses a method for tracking, locking and magnifying a portrait based on a conventional fixed-point fixed-focus camera. The method comprises the following steps: step 1, performing face detection by using a conventional fixed-point fixed-focus camera to obtain coordinate frames of all face regions in a video image; step 2, eliminating flickering of an external frame caused by false detection and missed detection of a face detection algorithm, and recording the number of times a face appears and disappears in consecutive frames by using a face marking method, thereby reducing the influence of fluctuations caused by missed detection and false detection; step 3, magnifying the face frame and calculating the external frame, first calculating a minimum external rectangular frame based on all face coordinate frames; step 4, performing Euro filtering and tracking processing on the external face frame, and filtering the calculated external frame using two different filtering algorithms; step 5, adjusting the aspect ratio of the external frame to be consistent with the aspect ratio of an input image; and step 6, performing gradual scaling on the image and filling a scaling queue.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for tracking, locking and magnifying a portrait based on a conventional fixed-point and fixed-focus camera. Background Art

[0002] Existing cameras capable of portrait tracking generally offer adjustable lens focal length and angle. These devices are complex, expensive, and relatively expensive to manufacture. They achieve face tracking through micro-motors controlling lens movement and focal length. Currently, most fixed-point cameras, conventional cameras, and built-in cameras on laptops lack single-function features, lacking portrait tracking, locking, or zooming capabilities. This can easily lead to unused and wasted hardware. While a limited number of these cameras offer face detection capabilities, the vast majority lack portrait tracking, locking, or dynamic zooming. The present invention can reuse these existing devices, improving their functionality. Summary of the Invention

[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention discloses a method for tracking, locking, and zooming in on a person's portrait based on a conventional fixed-point fixed-focus camera, the method comprising the following steps:

[0004] Step 1: Perform face detection using a conventional fixed-point fixed-focus camera to obtain the coordinate frames of all face regions in the video image;

[0005] Step 2: Eliminate the external frame flickering caused by false detection and missed detection of the face detection algorithm. Use the face marking method to record the number of times a face appears and disappears in consecutive frames, thereby reducing the fluctuation effect caused by missed detection and false detection.

[0006] Step 3: Enlarge the face frame and calculate the bounding box. First, calculate the minimum bounding rectangle based on all face coordinate frames.

[0007] Step 4: Euro filtering and tracking are performed on the external face frame, and two different filtering algorithms are used to filter the calculated external frame;

[0008] Step 5: Adjust the aspect ratio of the bounding box to be consistent with the aspect ratio of the input image;

[0009] Step 6: Gradual scaling of the image and filling the scaling queue.

[0010] Furthermore, the face region coordinate frame in step 1 includes the coordinates of the upper left corner, the width and height of the face frame, and the confidence of the face frame.

[0011] Furthermore, the step 2 further includes: in the face detection process, due to the camera clarity and the accuracy of the algorithm model, there is always a frame that is missed or misdetected in the continuous frames, which gradually causes a stable external frame mutation. First, all the faces that appear in the frame are recorded, and then the number of faces that appear in each frame is continuously counted in the next continuous frames.

[0012] The number of times a face appears and is lost is used to calculate the IOU method to determine which face in the statistics the face belongs to. Faces that appear more than a certain number of times are added to the calculation scope of the external frame, and faces that are lost a certain number of times are deleted from the statistics. This method can effectively reduce the impact of false detection and missed detection of face algorithms.

[0013] Furthermore, step 3 further includes: calculating the minimum bounding rectangle as follows: traversing all face coordinate frames, enlarging the size of the coordinate frame by a certain multiple so that it can cover most areas of the portrait, and finding the minimum value of the upper left corner coordinate and the maximum value of the lower right corner coordinate.

[0014] Furthermore, step 4 further includes: first, using a Euro filtering method to de-jitter the upper left corner and the lower right corner of the external rectangular box to obtain new coordinate points, and then using the SORT target tracking algorithm, using Kalman filtering to predict the rectangular box and combining the Hungarian algorithm to perform target association matching through the IOU method to effectively reduce the jitter problem of the target box.

[0015] Furthermore, the step 5 further includes: first using the greatest common divisor to calculate the aspect ratio of the input image; then filling the width and height of the external frame to be divisible by the aspect ratio of the original image, assuming the aspect ratio is N:M, first taking the modulus of the width and N, and then adding the value after subtracting the modulus from N to the width; dividing the filled width by N to obtain the multiple of the width, and dividing the filled height by M to obtain the multiple of the height, and calculating the maximum value of the two (so that the new width and height can wrap the external frame), and then taking the minimum value of the maximum value and the multiple of the original image (to avoid exceeding the image range) to obtain a suitable multiple value; multiplying M and N by the multiple value respectively to obtain the new width and height of the external frame, and the aspect ratio calculated in this way can ensure that it will be consistent with the original aspect ratio of the image, and the boundary will not exceed the original image.

[0016] Furthermore, N is set to 16 and M is set to 9.

[0017] Furthermore, step 6 further includes: gradual scaling of the image size, that is, the process of scaling the image to the target size frame by frame. The scaling method is to first determine N frames to be used for frame-by-frame scaling, then calculate the N position points and scaling coefficients connecting the current position and the target position, store the image area coordinates of the N frames that need to be changed in a queue, and finally dequeue them frame by frame from the queue to complete the frame-by-frame scaling process.

[0018] The present invention further discloses a computer-readable storage medium storing program codes, which implement the above method when executed by a processor.

[0019] The present invention further discloses a computing device, comprising a processor and a storage medium storing program code, wherein the program code, when executed by the processor, implements the method as described above.

[0020] Compared with the existing technology, the present invention has the following advantages: the present invention uses a low-cost fixed-point camera to track and lock the face and dynamically zoom the face area image, always keeping the entire face image in the zoomed area. The present invention uses a conventional fixed-point camera, USB camera, laptop camera and other portable device cameras to simulate the face tracking effect of a mobile camera and can highlight and magnify the face.

[0021] Continuously track faces. Using a fixed-point camera combined with algorithms and software, you can achieve face tracking and dynamic magnification of face areas at a low cost.

[0022] The general effect is described as follows:

[0023] 1. Portrait calculation lock: When one or more faces are present in the image area returned by the camera, a face area containing all faces is calculated and enlarged to the original image size to highlight the face area.

[0024] 2. Real-time face tracking: When a face moves in the input image, the detected face is continuously tracked and the position of the face area is dynamically adjusted to always capture the entire face image and keep it in a prominent position in the image.

[0025] 3. Suppress missed or false face detections. This can effectively reduce missed or false face detections caused by algorithmic issues.

[0026] 4. Use frame-by-frame gradual scaling. When scaling the face area, use frame-by-frame gradual scaling to reduce the abruptness caused by sudden scaling.

[0027] 5. Image display reset: When there is no face in the image, the image will be gradually reduced to its original size.

[0028] The face detection algorithm uses MNN as the inference engine when using CPU inference. This allows for pure CPU inference on most low-computing devices and is suitable for most devices. It can also select an appropriate inference backend when using an NPU or other GPU inference backend. The algorithm model uses 256x256, 128x128, and 128x96 input sizes, significantly reducing the model's computational complexity while maintaining a certain level of accuracy.

[0029] The use of the present invention requires access to video data, which can be a conventional camera or a real-time video stream. After the data is accessed, it is processed and analyzed according to the method of the present invention to obtain the processed image. The image frame data can be used for display or pushed to other processing channels.

[0030] According to the above model, you only need an ordinary camera to experience the portrait tracking and locking effect that only mobile cameras can achieve, which greatly improves the functionality of ordinary cameras and popularizes AI algorithms in daily usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention can be further understood from the following description in conjunction with the accompanying drawings. The components in the figures are not necessarily drawn to scale, but rather emphasis is placed on illustrating the principles of the embodiments. In the figures, the same reference numerals designate corresponding parts in different views.

[0032] Figure 1 This is a simplified processing flow chart of a method for portrait tracking, locking, and portrait magnification based on a conventional fixed-point and fixed-focus camera of the present invention;

[0033] Figure 2 This is a flow chart of a method for filtering missed and false face detections in one embodiment of the present invention;

[0034] Figure 3 It is a detailed processing flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment relies on a face detection algorithm and uses a face marking method to reduce the impact of missed detection and false detection. It uses a multi-filtering algorithm to smooth the external face frame and uses an image gradient scaling method to scale the image frame by frame to reduce the abruptness caused by portrait scaling. The specific method and process are as follows:

[0037] 1. Face Detection

[0038] The present invention relies on a face detection algorithm to obtain the coordinate frames of all face regions in a video image by performing face detection. The coordinate frames of the face regions include the coordinates of the upper left corner, the width and height of the face frame, and the confidence level of the face frame.

[0039] 2. Eliminate the problem of external frame flickering caused by false detection and missed detection of face detection algorithm

[0040] During the face detection process, due to camera clarity and algorithm model accuracy issues, there is always a frame that is missed or falsely detected in consecutive frames, which gradually causes a stable external frame mutation (the external frame becomes smaller due to missed face detection, or becomes larger due to false detection).

[0041] This paper uses a face tagging method to record the number of times a face appears and disappears in consecutive frames, thereby reducing the impact of fluctuations caused by missed and false detections. First, all faces appearing in a frame are recorded. Then, in subsequent consecutive frames, the number of times each face appears and disappears is continuously counted. An IOU calculation method is used to determine which face in the statistics the face belongs to. Faces that appear more than a certain number of times are added to the calculation scope of the bounding box, and faces that disappear a certain number of times are removed from the statistics. Using this method, the impact of false detection and missed detection caused by face algorithms can be effectively reduced.

[0042] The process flow chart is shown in Figure 2 .

[0043] 3. Face frame enlargement and external frame calculation

[0044] First, calculate the minimum bounding rectangle based on all face coordinate frames.

[0045] The calculation method is as follows: traverse all face coordinate frames, enlarge the size of the coordinate frame by a certain multiple so that it can cover most of the area of the portrait, and find the minimum value of the upper left corner coordinate and the maximum value of the lower right corner coordinate:

[0046] Upper left corner coordinate tl=()

[0047] Lower right corner coordinate br=()

[0048] for box in boxes:

[0049] tl=min(box.tl,tl)

[0050] br=max(box.br,br)

[0051] Minimum bounding rectangle: (tl,br)

[0052] 4. Perform Euro filtering and tracking on external face frames

[0053] In order to avoid shaking of the cropped human photo frame, the present invention uses two different filtering algorithms to filter the calculated external frame during the process.

[0054] The first step is to use the Euro filter method to de-jitter the upper left and lower right corners of the external rectangular box to obtain new coordinate points. The second step is to use the SORT target tracking algorithm, which mainly uses the Kalman filter to predict the rectangular box and combines it with the Hungarian algorithm to perform target association matching through the IOU method, which can also effectively reduce the jitter problem of the target box.

[0055] 5. Adjust the aspect ratio of the external frame to match that of the input image;

[0056] For example, the common head image size is 16:9. Since the person in the camera is constantly changing, the size of the calculated external frame is also constantly changing. Therefore, adjusting the width and height of the external frame to be consistent with the original image can avoid the stretching effect caused by image scaling. Common scaling methods are as follows:

[0057] W=16.0 / 9.0*h

[0058] H=w*9.0 / 16.0

[0059] Since computers process floating-point numbers with errors and need to convert floating-point numbers into integers, the final calculated aspect ratio always has errors compared to the original image. This invention uses a new calculation method to ensure the aspect ratio of the external frame:

[0060] First determine the aspect ratio of the input image, such as 4:3, 16:9, etc., or calculate it using the greatest common divisor.

[0061] Then, the width and height of the bounding box are filled to be divisible by the aspect ratio of the original image. Taking the width as an example, assuming the aspect ratio is 16:9, first find the modulus of the width and 16, and then add the value after subtracting the remainder from 16 to the width.

[0062] Divide the completed width by 16 to get the multiple of the width, and divide the completed height by 9 to get the multiple of the height. Calculate the maximum value of the two (so that the new width and height can wrap the external frame), and then minimize the maximum value and the multiple of the original image (to avoid exceeding the image range) to get the appropriate multiple value.

[0063] Multiply 16 and 9 by the multiple value respectively to obtain the new width and height of the bounding box.

[0064] The calculated aspect ratio is guaranteed to match the original image aspect ratio and the borders will not extend beyond the original image.

[0065] 6. Image gradual scaling and filling the scaling queue

[0066] Gradual image scaling, the process of scaling an image frame by frame toward the target size, provides a better visual experience and avoids the abrupt effects of a large difference from the target size. The scaling method first determines N frames to use for frame-by-frame scaling, then calculates the N points and scaling factors along the line connecting the current and target positions. The coordinates of the image regions of the N frames to be scaled are queued, and finally, the image regions are dequeued frame by frame to complete the frame-by-frame scaling process.

[0067] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or apparatus that includes a series of elements includes not only those elements,

[0068] It also includes other elements not explicitly listed, or elements inherent to such process, method, product or device. In the absence of more restrictions, the elements defined by the phrase "including a..." do not exclude the existence of other identical elements in the process, method, product or device that includes the elements.

[0069] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] Although the present invention has been described above with reference to various embodiments, it should be understood that many changes and modifications may be made without departing from the scope of the present invention. Therefore, it is intended that the above detailed description is considered to be illustrative and not restrictive, and it should be understood that the following claims (including all equivalents) are intended to limit the spirit and scope of the present invention. These embodiments are understood to be merely illustrative of the present invention and not intended to limit the scope of protection of the present invention. After reading the content of the record of the present invention, the technical staff may make various changes or modifications to the present invention, and these equivalent variations and modifications fall within the scope defined by the claims of the present invention.

Claims

1. A method for tracking, locking, and zooming in on a person based on a conventional fixed-point and fixed-focus camera, characterized in that: The method comprises the following steps: Step 1: Perform face detection using a conventional fixed-point fixed-focus camera to obtain the coordinate frames of all face regions in the video image; Step 2, eliminating the flickering of the external frame caused by false detection and missed detection of the face detection algorithm, and recording the number of times a face appears and is lost in consecutive frames by a face marking method, wherein step 2 further includes: during the face detection process, first recording all faces appearing in the frame, continuously counting the number of times each face appears and is lost in the following consecutive frames, using the IOU calculation method to determine which face in the statistics the face belongs to, adding faces that appear more than a certain number of times to the calculation scope of the external frame, and deleting faces that are lost a certain number of times from the statistics. This method can effectively reduce the impact caused by false detection and missed detection of the face algorithm; Step 3: Enlarge the face frame and calculate the bounding box. First, calculate the minimum bounding rectangle based on all face coordinate frames. Step 4: Euro filtering and tracking the external face frame. Two different filtering algorithms are used to filter the calculated external frame. Step 4 further includes: first, using a Euro filtering method to de-jitter the upper left corner and lower right corner of the external rectangular frame to obtain new coordinate points, then using the SORT target tracking algorithm, using Kalman filtering to predict the rectangular frame, and then combining the Hungarian algorithm to perform target association matching through the IOU method to effectively reduce the jitter problem of the target frame; Step 5. Adjust the aspect ratio of the bounding box to be consistent with that of the input image. First, use the greatest common divisor to calculate the aspect ratio of the input image. Then, fill the width and height of the bounding box to be divisible by the aspect ratio of the original image. Let the aspect ratio be N:M. First, take the modulus of the width and N, and then add the value after subtracting the modulus from N to the width. Divide the filled width by N to get the multiple of the width, and divide the filled height by M to get the multiple of the height. Calculate the maximum value of the two, and then minimize the maximum value and the multiple of the original image to get the appropriate multiple value. Multiply M and N by the multiple value to obtain the new width and height of the bounding box. The aspect ratio calculated in this way can ensure that it is consistent with the original aspect ratio of the image, and the boundary will not exceed the original image. Step 6, perform gradual scaling on the image and fill the scaling queue and output the image.

2. The method for tracking, locking, and zooming in on a person based on a conventional fixed-point and fixed-focus camera according to claim 1, wherein: The face region coordinate frame in step 1 includes the coordinates of the upper left corner, the width and height of the face frame, and the confidence of the face frame.

3. The method for tracking, locking, and zooming in on a person based on a conventional fixed-point and fixed-focus camera according to claim 1, wherein: Step 3 further includes: calculating the minimum bounding rectangle as follows: traversing all face coordinate frames, enlarging the size of the coordinate frame by a certain multiple so that it can cover most areas of the portrait, and finding the minimum value of the upper left corner coordinate and the maximum value of the lower right corner coordinate.

4. The method for tracking, locking, and magnifying a person's portrait based on a conventional fixed-point and fixed-focus camera according to claim 3, wherein: The value of N is 16 and the value of M is 9.

5. The method for tracking, locking, and magnifying a person's portrait based on a conventional fixed-point and fixed-focus camera according to claim 1, wherein: The step 6 further includes: gradual image size scaling, that is, the process of scaling the image to the target size frame by frame. The scaling method is to first determine N frames to be used for frame-by-frame scaling, then calculate the N position points and scaling coefficients of the line connecting the current position and the target position, store the image area coordinates of the N frames that need to be changed in a queue, and finally dequeue the frames from the queue to complete the frame-by-frame scaling process.

6. A computer-readable storage medium storing program code, wherein when the program code is executed by a processor, the program code implements the method according to any one of claims 1 to 5.

7. A computing device comprising a processor and a storage medium storing program code, wherein the program code, when executed by the processor, implements the method according to any one of claims 1 to 5.

Images