Video labeling method, device, and storage medium

Abstract

The present disclosure provides a video labeling method and device and a storage medium, and relates to the technical field of image processing. The video labeling method of the present disclosure comprises: for each image frame processing period, determining the coordinates of a region of interest (ROI) in a first frame image through image recognition, and labeling the ROI in the first frame image and a second frame image; starting from taking the second frame image as a current image frame: determining the coordinates of a predicted ROI in a next frame image according to the coordinates of the labeled ROI in the current image frame and based on video encoding prediction; taking the next frame image as a current frame image, and labeling the ROI in the current frame image according to the coordinates of the predicted ROI until the current frame image is the last frame image of the current processing period. Such a method can reduce the consumption of computing power while ensuring the accuracy of ROI labeling.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a video annotation method, apparatus, and storage medium. Background Technology

[0002] ROI (Region of Interest) is a commonly used technique in intelligent video coding. It can improve the image quality of ROI areas by using pre-marked ROI areas (such as doors, windows, and corridors).

[0003] Improving the accuracy of ROIs in intelligent video coding through ROI macroblock offset prediction enables dynamic ROI annotation, which can enhance the detail of moving people or vehicles in the scene at the same bitrate. Summary of the Invention

[0004] One objective of this disclosure is to reduce computational power consumption while ensuring the accuracy of ROI labeling.

[0005] According to one aspect of some embodiments of this disclosure, a video annotation method is proposed, comprising: for each image frame processing cycle, determining the coordinates of a Region of Interest (ROI) in a first frame image by image recognition, and annotating the ROI in the first frame image and a second frame image; starting from the second frame image as the current image frame: determining the coordinates of a predicted ROI in the next frame image based on the coordinates of the ROI annotated in the current image frame and a prediction based on video coding; taking the next frame image as the current frame image, and annotating the ROI in the current frame image according to the coordinates of the predicted ROI, until the current frame image is the last frame image of the current processing cycle.

[0006] In some embodiments, determining the coordinates of the predicted ROI in the next frame image based on the coordinates of the ROI marked in the current image frame and the prediction based on video coding includes: determining the macroblock offset coordinates and offset vector of the macroblock corresponding to the ROI in the previous frame image in the current frame image based on video coding; and determining the coordinates of the predicted ROI in the next frame image based on the coordinates of the ROI marked in the current image frame, as well as the macroblock offset coordinates and offset vector.

[0007] In some embodiments, the video annotation method further includes: after determining the coordinates of the ROI in the first frame image, encoding the first frame image by a video encoder; determining the macroblock offset coordinates and offset vector of the macroblock corresponding to the ROI in the previous frame image in the current frame image based on the video encoding includes: encoding the current frame image by a video encoder; determining the macroblock offset coordinates and offset vector of the macroblock in the current frame image based on the encoding of the current frame image and the encoding of the previous frame image.

[0008] In some embodiments, determining the macroblock offset coordinates and offset vector in the current frame image based on the encoding of the current frame image and the encoding of the previous frame image includes: determining the four corner coordinates of the macroblock in the current frame image based on the output data of the video encoder; determining the minimum bounding rectangle of the macroblock based on the four corner coordinates, and using the coordinates of the four corners of the minimum bounding rectangle as the macroblock offset coordinates; and determining the macroblock offset vector based on the coordinate difference between the center point of the four corner coordinates of the macroblock in the current frame image and the center point of the ROI marked in the previous frame image.

[0009] In some embodiments, the video annotation method further includes: after annotating the ROI region of the current frame image, performing YUV image encoding on the RAW image based on the ROI region, wherein the current frame image includes the image from the first frame image to the penultimate frame image of the current processing cycle; encoding the current frame image by the video encoder includes: inputting the YUV image of the current frame into the video encoder to encode the current frame image.

[0010] In some embodiments, determining the coordinates of the predicted ROI in the next frame image based on the macroblock offset coordinates and offset vector in the current frame image includes: determining the ROI region change factor based on the number of pixels in the region surrounded by the marked ROI in the current frame image and the region surrounded by the macroblock offset coordinates; determining the offset center coordinates based on the center coordinates of the region surrounded by the marked ROI in the current frame image and the macroblock offset vector in the current frame image; using the offset center coordinates as the center coordinates of the predicted ROI in the next frame image, adjusting the size of the region surrounded by the marked ROI in the current frame image according to the ROI region change factor, and determining the coordinates of the predicted ROI in the next frame image.

[0011] In some embodiments, determining the coordinates of a Region of Interest (ROI) in a first frame image by image recognition includes: performing YUV image encoding on the first frame image in RAW format to obtain a first frame YUV image; performing image recognition on the first frame YUV image to obtain the location of a target in the image and determine the ROI in the first frame YUV image; and marking the ROI in the first frame YUV image based on the ROI in the first frame YUV image.

[0012] In some embodiments, the video annotation method further includes: acquiring a real-time image frame; determining whether the real-time image frame is the first frame image of the current processing cycle; if the real-time image frame is the first frame image of the current processing cycle, performing an operation to determine the coordinates of the ROI in the first frame image through image recognition; otherwise, performing an operation to determine the coordinates of the predicted ROI in the next frame image.

[0013] According to one aspect of some embodiments of this disclosure, a video annotation apparatus is proposed, comprising: an identification unit configured to determine the coordinates of a Region of Interest (ROI) in a first frame image for each processing cycle of an image frame by image recognition; an annotation unit configured to annotate the ROI in the first and second frames images according to the coordinates determined by the identification unit, and to annotate the ROI in the third to last frames images of the processing cycle according to the coordinates of the predicted ROI; and a prediction unit configured to, for each processing cycle, starting from the second frame image as the current image frame: after the annotation unit annotates the ROI in the current image frame, determine the coordinates of the predicted ROI in the next frame image according to the coordinates of the ROI annotated in the current image frame and inter-frame prediction based on video coding; and to use the next frame image as the current frame image and trigger the annotation unit to perform annotation, until the current frame image is the last frame image of the current processing cycle.

[0014] In some embodiments, the video annotation apparatus further includes: a video encoder configured to perform image encoding on the input image to determine the macroblock offset coordinates and offset vector of the macroblock corresponding to the ROI in the previous frame image in the current frame image.

[0015] In some embodiments, the video annotation apparatus further includes: a format conversion unit configured to perform YUV image encoding on the input RAW image; and a video encoder configured to perform image encoding on the input YUV image annotated with ROI.

[0016] According to one aspect of some embodiments of this disclosure, a video annotation apparatus is proposed, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute any of the video annotation methods described above based on instructions stored in the memory.

[0017] According to one aspect of some embodiments of this disclosure, a non-transitory computer-readable storage medium is proposed, having stored thereon computer program instructions that, when executed by a processor, implement the steps of any of the video annotation methods described above. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0019] Figure 1 Flowcharts showing some embodiments of the video annotation method disclosed herein.

[0020] Figure 2 Flowcharts for some other embodiments of the video annotation method disclosed herein.

[0021] Figure 3Flowcharts of some further embodiments of the video annotation method disclosed herein.

[0022] Figure 4 Flowcharts of further embodiments of the video annotation method disclosed herein.

[0023] Figure 5 These are schematic diagrams of some embodiments of the video annotation apparatus disclosed herein.

[0024] Figure 6 These are schematic diagrams of other embodiments of the video annotation apparatus of this disclosure.

[0025] Figure 7 This is a schematic diagram of some further embodiments of the video annotation apparatus of this disclosure. Detailed Implementation

[0026] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments.

[0027] The inventors discovered that intelligent video encoding in related technologies, which identifies moving objects in the image before labeling and encoding the Region of Interest (ROI), has the following problems:

[0028] 1. High computing power requirement: The method of recognizing moving objects in each frame of the image in related technologies requires a lot of computing power, which is generally not supported by cameras or reduces the encoding efficiency.

[0029] 2. ROI region delay: In related technologies, only one frame of image is selected for recognition and ROI labeling per k frames. For the other k-1 frames besides the first frame, the position of the ROI region is often delayed compared to the actual image. Especially for fast-moving people and vehicles, the labeling position is inaccurate, resulting in poor image optimization.

[0030] Flowcharts of some embodiments of the video annotation method disclosed herein are as follows: Figure 1 As shown.

[0031] In step 120, the coordinates of the Region of Interest (ROI) in the first frame image are determined by image recognition, and the ROI is labeled in the first frame image. In some embodiments, relevant feature parameters of the target can be set, and the area occupied by the target in the image can be determined by image recognition algorithms in related technologies. The area occupied by the target in the image is then used as the ROI and labeled. In some embodiments, the labeling method can be to improve the clarity of the ROI, thereby preserving the image details of the ROI. In some embodiments, the labeling method can also be to select the ROI by drawing a box in the image, thereby facilitating the user to quickly identify and lock the target. In some embodiments, the target can be a person, a vehicle, or other objects set as needed.

[0032] In some embodiments, images can be acquired from the image acquisition section of an image sensor, such as a camera or camcorder. In some embodiments, considering that the acquired images may use image formats such as RAW, which are raw data from the sensor converting captured light source signals into digital signals, making processing difficult, the operation of identifying the ROI in the first frame image can be performed as follows: Figure 2 As shown in the image.

[0033] In step 221, the first frame of the RAW format image is encoded using YUV image encoding to obtain the first frame of YUV image. YUV format is a method of image encoding that includes luminance (grayscale) and chrominance information, and its data volume is significantly reduced compared to RAW format.

[0034] In step 222, image recognition is performed on the first frame YUV image to obtain the location of the target in the image and determine the ROI in the first frame YUV image.

[0035] In step 223, the ROIs are marked in the first frame of the RAW format image based on the ROIs in the first frame of the YUV image.

[0036] Based on the method in this embodiment, the RAW format image can be converted to obtain the required YUV format image with low computing power and high processing efficiency. Then, ROI identification can be performed in the YUV format image, which reduces the performance requirements of the image processing equipment and improves the identification efficiency. Marking the identified ROI in the original RAW format image avoids information loss and further ensures the image accuracy.

[0037] In step 130, the second frame image is used as the current frame image, and the ROI is labeled. The coordinates of the labeled ROI are the same as those in the first frame image.

[0038] In step 140, the coordinates of the predicted ROI in the next frame are determined based on the coordinates of the ROI marked in the current image frame and the prediction based on video coding.

[0039] In some embodiments, taking the current image frame as the second frame for example, the prediction ability of the encoder (such as inter-frame prediction) can be utilized to obtain the change trend of the macroblock where the ROI is located (including the change trends of the central position and size) based on the first frame image and the second frame image, and then this trend is used as the basis for predicting the ROI in the next frame image. Similarly, taking the current image as the nth frame image for example (2 < n < the number of image frames in a single processing cycle), the change trend of the macroblock where the ROI is located is obtained based on the nth frame image and the previous images, and then this trend is used as the basis for predicting the ROI in the (n + 1)th frame image. In some embodiments, the previous images mentioned above can specifically refer to the previous frame image, that is, the (n - 1)th frame image, or can be the images from the 1st frame to the (n - 1)th frame, which is determined by the prediction algorithm set by the encoder.

[0040] In step 150, the next frame image is taken as the current frame image. In some embodiments, if the current frame image is the last frame image, that is, there is no next frame image, the process ends.

[0041] In step 160, the ROI is labeled in the current frame image according to the coordinates of the predicted ROI. In some embodiments, the labeling method can be to improve the clarity of the ROI.

[0042] In step 170, it is judged whether the current frame image is the last frame image of the current processing cycle. If it is the last frame image, there is no next frame image in the current processing cycle, and step 180 is executed; otherwise, return to execute step 140.

[0043] In step 180, the current processing cycle ends. In some embodiments, if the current processing cycle is not the last processing cycle, return to step 120 to execute the recognition of the first frame image of the next processing cycle; otherwise, the image processing ends.

[0044] Based on the method in the above embodiments, it is possible to perform ROI prediction by using macroblock offset analysis in coding. Compared with the method of identifying ROI at intervals in the related art, it can improve the accuracy of ROI labeling without additional computing power, and from another perspective, it can also reduce the computing power consumption on the basis of ensuring the accuracy of ROI labeling.

[0045] In some embodiments, the change trend of the macroblock can be represented by parameters in two aspects: the macroblock offset coordinates and the offset vector. Among them, the macroblock offset coordinates refer to the coordinates of the macroblock corresponding to the ROI in the previous frame image in the current frame image, which can be represented by the four corner coordinates of the macroblock area, and the offset vector refers to the displacement of the center of the macroblock area compared with the center in the previous frame image, which can be represented by a vector (x, y) containing two parameters in the horizontal and vertical directions.

[0046] In some embodiments, since the area enclosed by the four corner coordinates of the offset macroblock may not be rectangular, considering the performance and compatibility of the image processing device, the minimum bounding rectangle of the corresponding area can be taken based on the four corner coordinates obtained from video coding, and the four corner coordinates of this minimum bounding rectangle can be used as the macroblock offset coordinates. In some embodiments, the method for determining the coordinates of the predicted ROI in the next frame image in step 140 above can be as follows: Figure 3 As shown.

[0047] In step 341, the four corner coordinates of the macroblock in the current frame image are determined based on the output data of the video encoder. In some embodiments, the current frame image can be input into the video encoder, which has already received previous image frames, thereby enabling macroblock offset analysis. In some embodiments, in step 120 above, the first frame image with ROIs marked can be input into the video encoder as the data basis for determining the macroblock position in the second frame image.

[0048] In some embodiments, in order to improve image processing efficiency, the RAW format image with the predicted ROI marked can be converted to a YUV format image first, and the current frame image in YUV format can be input into the video encoder, thereby avoiding excessive processing pressure on the encoder and improving computational efficiency.

[0049] In step 342, the minimum bounding rectangle of the macroblock is determined based on the four corner coordinates, and the coordinates of the four corners of the minimum bounding rectangle are used as the macroblock offset coordinates.

[0050] In step 343, the macroblock offset vector is determined based on the difference between the coordinates of the center point of the quadrilateral region formed by the four corner coordinates of the macroblock in the current frame image (i.e., the four corner coordinates determined in step 341) and the coordinates of the center point of the ROI marked in the previous frame image.

[0051] In step 344, the ROI region change factor is determined based on the number of pixels in the region surrounded by the labeled ROI and the region surrounded by the macroblock offset coordinates in the current frame image.

[0052] In step 345, the offset center coordinates are determined based on the center coordinates of the region enclosed by the ROI marked in the current frame image and the offset vector of the macroblock in the current frame image. In some embodiments, the offset center coordinates are the coordinates of the center point of the quadrilateral region formed by using the four corner coordinates in step 341 as vertices.

[0053] In step 346, the offset center coordinates are used as the center coordinates of the predicted ROI in the next frame image. The size of the region enclosed by the marked ROI in the current frame image is adjusted according to the ROI region change factor to determine the coordinates of the predicted ROI in the next frame image. In some embodiments, the offset center coordinates are used as the center coordinates of the predicted ROI. The width and length of the region enclosed by the marked ROI in the current frame image are multiplied by the ROI region change factor to determine the width and length of the predicted ROI in the next frame image, thus obtaining the coordinates of the predicted ROI in the next frame image.

[0054] Based on the method in the above embodiments, the changes in macroblocks in the current frame image compared to the previous frame image can be determined through encoding. These changes are then used as a trend to determine the predicted ROI of the next frame image and to annotate the next frame image. Therefore, for non-first frame images in each processing cycle, ROI annotation can be achieved before the encoding operation of that frame, thereby improving the speed and real-time performance of ROI annotation in the image, reducing the latency between annotation and viewing, and improving annotation efficiency.

[0055] In some embodiments, image frames can be acquired in real time, such as by acquiring images through image acquisition devices like surveillance cameras, and then performing real-time processing. Figure 4 The processing in the illustrated embodiment improves the real-time performance of viewing images with ROI annotations. Flowcharts of some embodiments of the video annotation method disclosed herein are shown below. Figure 4 As shown.

[0056] In step 411, real-time image frames are acquired. In some embodiments, a starting point and an image processing cycle can be set, with each image processing cycle consisting of a predetermined number of k image frames, such as k = 10, 15, or 30. In some embodiments, each acquired real-time image frame can be numbered, with the number i starting from 0 and incrementing by one.

[0057] In step 412, it is determined whether the real-time image frame is the first frame of the current processing cycle. If it is the first frame, step 421 is executed; otherwise, step 431 is executed.

[0058] In some embodiments, it can be determined whether n%k (i.e., the remainder when n is divided by k) is 0. If n%k = 0, then the real-time image frame is the first frame of the current processing cycle; otherwise, the real-time image frame is not the first frame of the current processing cycle.

[0059] In step 421, the coordinates of the Region of Interest (ROI) in the first frame image are determined by image recognition and used as the predicted ROI for the second frame. The ROI is then labeled in the first frame image. In some embodiments, the coordinates of the ROI in the real-time image frame can be identified based on the method described in step 120 above, and labeled in the real-time image frame to facilitate real-time image display.

[0060] In step 422, the real-time image frame marked with ROI is input into the video encoder as the data basis for encoding and analyzing the macroblock offset of the next frame image.

[0061] In step 431, the predicted ROI is labeled in the current frame. In some embodiments, if n%k=1 in the operation of step 412 for the current real-time image frame, the predicted ROI is determined according to the operation in step 421 of the previous loop; otherwise, the predicted ROI is determined according to the operation in step 432 of the previous loop.

[0062] In step 432, the coordinates of the predicted ROI in the next frame are determined based on the coordinates of the ROI labeled in the current image frame and the prediction based on video coding. In some embodiments, this can be based on, for example... Figure 3 The method described in the embodiment is to determine the coordinates of the predicted ROI in the next frame image.

[0063] Based on the methods described in the above embodiments, continuous image ROI identification and prediction can be performed based on the acquired image frames, and annotation can be done in real time, improving the efficiency of providing ROI-annotated images and enhancing the user experience. In some embodiments, the annotated images can be stored in real time in storage devices (such as SD (Secure Digital) memory cards, NVRs (Network Video Recorders), etc.), and can also be sent to the business platform via the network for convenient use by the business platform.

[0064] The method described in the above embodiments can predict the ROI region through macroblock offset in video encoding, solving the problems of high computing power requirements and long processing time caused by target detection in each frame. This allows intelligent video encoding technology to be applied to low-computing-power and ordinary intelligent cameras, making AI (Artificial Intelligence) monitoring and other services more accessible. By using macroblock offset technology to dynamically predict and adjust the ROI region, it ensures that the ROI region can cover fast-moving people and vehicles in the video frame in real time, meeting users' requirements for detail in surveillance videos and television / movie content.

[0065] In some embodiments, the service platform can play the labeled images in real time, or retrieve and play images within a specified time period according to user instructions, allowing users to review them in the cloud, thereby improving the flexibility of image application.

[0066] In some embodiments, taking a predetermined number of image frames k = 30 in the processing cycle as an example, the encoding parameters are:

[0067] Resolution: 1920*1080

[0068] Graphics Group GOP: 30

[0069] Frame rate (FPS): 15

[0070] ROI QP: 22

[0071] QP for non-ROI regions: unROIQP: 37

[0072] Image recognition is performed only on keyframes, i.e., once every 30 frames, and the ROI region is adjusted by prediction for the remaining 29 frames.

[0073] 1. Image encoding of the first frame (n%k=0):

[0074] 1.1. Acquire the first frame image in RAW format from the sensor and set n=0.

[0075] 1.2. The RAW image data is converted to YUV image data using an encoder, and image recognition is performed. The recognition results are as follows:

[0076] 1) The coordinates of the person being identified and the four corner coordinates of the corresponding ROI are:

[0077] [(300,300),(600,300),(300,700),(600,700)];

[0078] 2) The ROI region has (600-300)*(700-300)=120000 pixels, and the QP value is set to 22;

[0079] 3) The number of pixels in the non-ROI area is 1920*1080-120000=1953600, and the QP value is set to 37.

[0080] 1.3 After marking the ROI region of the RAW image, re-encode the RAW image using YUV image encoding, and import the YUV image encoding into the video encoder for encoding the first frame.

[0081] 1.4 The encoder outputs the ROI macroblock offset coordinates as the predicted ROI coordinates for the second frame image. Since it is the first frame, the ROI macroblock has no offset.

[0082] 1) The four corner coordinates of the ROI macroblock are [(300,300),(600,300),(300,700),(600,700)];

[0083] 2) The ROI macroblock offset vector is [0,0].

[0084] 2. Image encoding of the second frame (n%k=1):

[0085] 2.1 Obtain the second frame image in RAW format from the sensor and set n = n + 1.

[0086] 2.2 Since n%30! = 0, it is necessary to label the coordinates of the predicted ROI. For the second frame image, the predicted ROI is the ROI detected in the first frame image:

[0087] 1) The four corner coordinates of the ROI macroblock are [(300,300),(600,300),(300,700),(600,700)];

[0088] 2) The ROI macroblock offset vector is [0,0].

[0089] 3) Label the ROI region [(300,300),(600,300),(300,700),(600,700)].

[0090] 2.3 After marking the ROI region of the RAW image, re-encode the RAW image using YUV image encoding, and import the YUV image encoding into the video encoder for encoding the second frame image.

[0091] 2.4 The encoder outputs the offset coordinates of the ROI macroblock and calculates the ROI macroblock offset vector:

[0092] 1) The four corner coordinates after the ROI macroblock offset are:

[0093] [(300,300),(800,300),(300,800),(700,700)];

[0094] 2) The coordinates of the four corners after the rectangular ROI macroblock offset are:

[0095] [(300,300),(800,300),(300,800),(800,800)];

[0096] 3) Based on the four corner coordinates in 2.4-1) and the coordinates of the ROI region marked in the current image frame, calculate the ROI macroblock offset vector Q of frame n. n [x,y]:

[0097]

[0098] in, These are the coordinates of the top-left, bottom-left, top-right, and bottom-right corners of the ROI macroblock in the frame n image output by the encoder, respectively.

[0099] Taking n=1 as an example, based on the above data, the macroblock offset vector Q1[x,y]=[75,25] is obtained.

[0100] 3. From the third frame to the last frame in the cycle (n%k = 2~29)

[0101] 3.1 Obtain the RAW image of the current frame from the Sensor, n = n + 1.

[0102] 3.2 Since n%k! = 0, ROI region prediction is required: Taking the 3rd frame (n=2) as an example:

[0103] 1) The coordinates of the four corners of the ROI marked in the second frame image above are:

[0104] [(300,300),(600,300),(300,700),(600,700)], center coordinates (450,500), ROI region pixels = 120000

[0105] 2) The four corner coordinates of the ROI macroblock offset in the second frame are:

[0106] [(300,300),(800,300),(300,800),(800,800)], pixels enclosed by macroblock offset.

[0107] =250000

[0108] 3) The predicted ROI region in the 3rd frame is larger.

[0109]

[0110] 4) Based on the ROI macroblock offset vector [75,25] in the second frame, predict the center coordinates of the third frame as (450+75,500+25)=(525,525).

[0111] 5) Based on the region scaling factor and center coordinates, calculate the four corner coordinates of the ROI region in the third frame: [(275,275),(775,275),(275,775),(775,775)]

[0112] 6) Mark the ROI region of the current frame.

[0113] 3.3. Perform YUV image encoding on the RAW image based on the ROI region, and import the YUV image encoding into the video encoder to encode the (n-1)th frame image;

[0114] 3.4 Using the method described in 2.4 above, the encoder outputs the ROI macroblock offset coordinates of the (n-1)th frame image, calculates the ROI macroblock offset vector, and continues the cyclic encoding.

[0115] The specific figures in the above embodiments are merely examples and do not constitute undue limitation on this application. Based on this approach, the accuracy of ROI labeling can be improved through video coding macroblock offset prediction, enhancing the universality of intelligent video coding and improving video image quality and detail while ensuring video bandwidth.

[0116] Schematic diagrams of some embodiments of the video annotation device disclosed herein are shown below. Figure 5 As shown.

[0117] The recognition unit 501 is capable of determining the coordinates of the Region of Interest (ROI) in the first frame image through image recognition for each image frame processing cycle. In some embodiments, the recognition unit 501 can be based on, for example... Figure 2 The method used in the illustrated embodiment is to identify and label ROIs. In some embodiments, the identification unit 501 may call the format conversion unit 505 to perform RAW to YUV image format conversion.

[0118] The annotation unit 502 can annotate the ROIs in the first and second frame images according to the coordinates determined by the recognition unit, and annotate the ROIs in the third to last frame images in the processing cycle according to the predicted coordinates of the ROIs. In some embodiments, the annotation unit 502 can provide the ROI with a higher QP (Quantizer Parameter) than other parts of the image to improve the clarity of the ROI image.

[0119] The prediction unit 503, for each processing cycle, starts by using the second frame as the current image frame: after the annotation unit annotates the ROI in the current image frame, it determines the coordinates of the predicted ROI in the next frame image based on the coordinates of the annotated ROI in the current image frame and inter-frame prediction based on video coding; it then uses the next frame image as the current frame image and triggers the annotation unit to perform annotation, until the current frame image is the last frame image of the current processing cycle. In some embodiments, the prediction unit 503 can be based on, for example, Figure 1 The ROI is predicted using the methods described in steps 140-180.

[0120] The video annotation device in the above embodiment can use macroblock offset analysis in the encoding to predict ROI. Compared with the intermittent ROI identification method in related technologies, it can improve the accuracy of ROI annotation without increasing computing power. From another perspective, it can also reduce computing power consumption while ensuring the accuracy of ROI annotation.

[0121] In some embodiments, such as Figure 5 As shown, the video annotation device may also include a video encoder 504, which can encode the input image. For example, the encoder decoder uses encoding algorithms such as H.265 and H.264 to determine the macroblock offset coordinates and offset vector of the macroblock corresponding to the ROI in the previous frame image in the current frame image, and encodes the image into video. This fully utilizes the information generated during the encoding process to compress the data volume, which can reflect the differences between frames, and uses it as the data basis for predicting the ROI in the next frame. This improves the accuracy of ROI annotation without putting computational pressure on the device.

[0122] In some embodiments, such as Figure 5 As shown, the video annotation apparatus may further include a format conversion unit 505, capable of performing YUV image encoding on the input RAW image. Thus, the video encoder 504 processes YUV images during image encoding. Such a video annotation apparatus can reduce computational load and increase processing speed. In some embodiments, the format conversion unit 505 can perform ISP (Image Signal Processing) on ​​the RAW image. If the ROI (Region of Interest) is already annotated in the RAW image, it reduces the QP (Quick Point Rate) for specific areas of the image based on the ROI annotation information, allocates a higher bitrate to improve image quality, and outputs YUV image data. This reduces the amount of data transmitted and improves transmission efficiency while ensuring image quality in key areas.

[0123] A schematic diagram of one embodiment of the video annotation device disclosed herein is shown below. Figure 6 As shown, the video annotation apparatus includes a memory 601 and a processor 602. The memory 601 can be a disk, flash memory, or any other non-volatile storage medium. The memory stores instructions in the corresponding embodiments of the video annotation method described above. The processor 602 is coupled to the memory 601 and can be implemented as one or more integrated circuits, such as a microprocessor or microcontroller. The processor 602 executes the instructions stored in the memory, enabling reduced computational power consumption while ensuring the accuracy of ROI annotation.

[0124] In one embodiment, it can also be as follows: Figure 7As shown, the video annotation device 700 includes a memory 701 and a processor 702. The processor 702 is coupled to the memory 701 via a BUS bus 703. The video annotation device 700 can also be connected to an external storage device 705 via a storage interface 704 to access external data, and can also be connected to a network or another computer system (not shown) via a network interface 706. Further details are omitted here.

[0125] In this embodiment, by storing data instructions in a memory and then processing the instructions by a processor, the computational power consumption can be reduced while ensuring the accuracy of ROI labeling.

[0126] In another embodiment, a computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the steps of the method in the corresponding embodiment of the video annotation method. Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0128] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0130] This concludes the detailed description of the present disclosure. To avoid obscuring the concept of the disclosure, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions disclosed herein based on the above description.

[0131] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit them; although this disclosure has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of this disclosure or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of this disclosure, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in this disclosure.

Claims

1. A video annotation method, comprising: For each image frame processing cycle, The coordinates of the region of interest (ROI) in the first frame image are determined by image recognition, and the ROI is marked in the first frame image and the second frame image. Starting with the second frame as the current frame: Based on the coordinates of the ROI labeled in the current image frame and the prediction based on video coding, the coordinates of the predicted ROI in the next image frame are determined as follows: Determining the macroblock offset coordinates and offset vector of the macroblock corresponding to the ROI in the previous frame image based on video encoding includes: determining the four corner coordinates of the macroblock in the current frame image based on the output data of the video encoder; using the four corner coordinates of the macroblock in the current frame image as the macroblock offset coordinates, or using the coordinates of the four corners of the minimum bounding rectangle of the macroblock determined based on the four corner coordinates as the macroblock offset coordinates; and determining the macroblock offset vector based on the coordinate difference between the center point of the four corner coordinates of the macroblock in the current frame image and the center point of the ROI marked in the previous frame image. Determining the coordinates of the predicted ROI in the next frame image based on the coordinates of the ROI marked in the current image frame, the macroblock offset coordinates, and the offset vector includes: determining the ROI region change factor based on the number of pixels in the region surrounded by the marked ROI in the current frame image and the region surrounded by the macroblock offset coordinates; determining the offset center coordinates based on the center coordinates of the region surrounded by the marked ROI in the current frame image and the macroblock offset vector in the current frame image; using the offset center coordinates as the center coordinates of the predicted ROI in the next frame image; adjusting the size of the region surrounded by the marked ROI in the current frame image according to the ROI region change factor; and determining the coordinates of the predicted ROI in the next frame image. The next frame image is used as the current frame image, and the ROI is labeled in the current frame image according to the coordinates of the predicted ROI, until the current frame image is the last frame image of the current processing cycle.

2. The method according to claim 1, further comprising: After determining the coordinates of the ROI in the first frame image, the first frame image is encoded by a video encoder.

3. The method according to claim 2, further comprising: After annotating the ROI region of the current frame image, the RAW image is YUV encoded based on the ROI region. The current frame image includes the image from the first frame image to the second to last frame image in the current processing cycle. The process of encoding the current frame image using a video encoder includes: inputting the YUV image of the current frame into the video encoder to encode the current frame image.

4. The method according to claim 1, wherein, The process of determining the coordinates of the ROI in the first frame image through image recognition includes: Perform YUV image encoding on the first frame of the RAW format image to obtain the first frame of the YUV image; Image recognition is performed on the first frame of YUV image to obtain the location of the target in the image and determine the ROI in the first frame of YUV image; Based on the ROI in the first frame YUV image, mark the ROI in the first frame RAW format image.

5. The method according to any one of claims 1 to 4, further comprising: Acquire real-time image frames; Determine whether the real-time image frame is the first frame of the current processing cycle; If the real-time image frame is the first frame of the current processing cycle, then the operation of determining the coordinates of the ROI in the first frame image through image recognition is performed; Otherwise, perform the operation to determine the coordinates of the predicted ROI in the next frame image.

6. The method according to any one of claims 1 to 4, further comprising: Store the image after ROI labeling and send it to the business platform via the network.

7. The method according to claim 6, further comprising: The business platform can play the labeled images in real time, or retrieve and play images within a specified time period according to user instructions.

8. A video annotation device, comprising: The recognition unit is configured to determine the coordinates of the region of interest (ROI) in the first image frame through image recognition for each processing cycle of the image frame. The annotation unit is configured to annotate ROIs in the first frame image and the second frame image according to the coordinates determined by the recognition unit, and to annotate ROIs in the third to last frame images in the processing cycle according to the coordinates of the predicted ROIs. The prediction unit is configured as follows: For each processing cycle, starting with the second frame as the current image frame: after the annotation unit annotates the ROI in the current image frame, it determines the coordinates of the predicted ROI in the next image frame based on the coordinates of the annotated ROI in the current image frame and the inter-frame prediction based on video coding, including: Determining the macroblock offset coordinates and offset vector of the macroblock corresponding to the ROI in the previous frame image based on video encoding includes: determining the four corner coordinates of the macroblock in the current frame image based on the output data of the video encoder; using the four corner coordinates of the macroblock in the current frame image as the macroblock offset coordinates, or using the coordinates of the four corners of the minimum bounding rectangle of the macroblock determined based on the four corner coordinates as the macroblock offset coordinates; and determining the macroblock offset vector based on the coordinate difference between the center point of the four corner coordinates of the macroblock in the current frame image and the center point of the ROI marked in the previous frame image. Determining the coordinates of the predicted ROI in the next frame image based on the coordinates of the ROI marked in the current image frame, the macroblock offset coordinates, and the offset vector includes: determining the ROI region change factor based on the number of pixels in the region surrounded by the marked ROI in the current frame image and the region surrounded by the macroblock offset coordinates; determining the offset center coordinates based on the center coordinates of the region surrounded by the marked ROI in the current frame image and the macroblock offset vector in the current frame image; using the offset center coordinates as the center coordinates of the predicted ROI in the next frame image; adjusting the size of the region surrounded by the marked ROI in the current frame image according to the ROI region change factor; and determining the coordinates of the predicted ROI in the next frame image. The next frame image is used as the current frame image, and the annotation unit is triggered to perform annotation until the current frame image is the last frame image of the current processing cycle.

9. The apparatus according to claim 8, further comprising: The video encoder is configured to encode the input image and determine the macroblock offset coordinates and offset vector of the macroblock corresponding to the ROI in the previous frame image in the current frame image.

10. The apparatus according to claim 8, further comprising: The format conversion unit is configured to perform YUV image encoding on the input RAW image; The video encoder is configured to perform image encoding on input YUV images labeled with ROIs.

11. A video annotation device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to perform the method as described in any one of claims 1 to 7 based on instructions stored in the memory.

12. A non-transitory computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images