Image processing device, image processing method, and image processing program

The video processing device corrects gaze area predictions based on object detection results to address image quality issues, achieving low-delay and high-quality video processing by using the corrected ROI for encoding.

WO2024247646A1PCT designated stage expired Publication Date: 2024-12-05NEC CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/017286
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-06-02
Filing Date
2024-05-09
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing video processing technologies face challenges in suppressing image quality deterioration due to incorrect region of interest (ROI) prediction, which can lead to delays and poor encoding performance.

Method used

A video processing device and method that predicts a gaze area in a target frame based on object detection results from a previous frame, corrects the prediction using object detection and prediction results, and sets the corrected gaze area for the target frame, thereby reducing delays and maintaining image quality.

Benefits of technology

This approach effectively suppresses image quality deterioration while enabling low-delay video processing by correcting ROI predictions and using the corrected gaze area for encoding, ensuring efficient video processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024017286_05122024_PF_FP_ABST
    Figure JP2024017286_05122024_PF_FP_ABST
Patent Text Reader

Abstract

This image processing device comprises: a prediction unit that, on the basis of an object detection result in an image of a frame which precedes a target frame, predicts a region of interest in an image of the target frame; a correction unit that, on the basis of an object prediction result and the object detection result in the image of the frame which precedes the target frame, corrects the region of interest predicted by the prediction unit; and a setting unit that sets the corrected region of interest as a region of interest with respect to the image of the target frame.
Need to check novelty before this filing date? Find Prior Art

Description

Video processing device, video processing method, and video processing program

[0001] The present disclosure relates to a video processing device, a video processing method, and a video processing program that set a gaze area and process video.

[0002] A video processing method using a region of interest (ROI) is known. For example, Patent Document 1 describes an image encoding device that encodes an ROI in an image. Patent Document 1 also describes that the motion of the ROI may be calculated using motion prediction of a cut-out image, and that the cut-out image is encoded using the motion prediction.

[0003] Japanese Patent Application Laid-Open No. 2021-166317

[0004] The image processing device described in Patent Document 1 reduces processing delays by predicting the ROI of a current frame based on information obtained from past frames. However, if the ROI prediction is incorrect, the image processing device may not be able to encode based on an appropriate ROI. In other words, if the ROI prediction is incorrect, image quality may be degraded.

[0005] Therefore, an object of the present disclosure is to provide a video processing device, a video processing method, and a video processing program that can suppress the occurrence of image quality degradation while performing low-latency video processing.

[0006] The video processing device according to the present disclosure includes a prediction means for predicting a gaze area in an image of a target frame based on an object detection result in an image of a frame prior to the target frame, a correction means for correcting the gaze area predicted by the prediction means based on the object detection result and the object prediction result in an image of a frame prior to the target frame, and a setting means for setting the corrected gaze area as the gaze area for the image of the target frame.

[0007] In the video processing method disclosed herein, a computer predicts a gaze area in an image of a target frame based on an object detection result in an image of a frame prior to the target frame, corrects the predicted gaze area based on the object detection result and object prediction result in the image of the frame prior to the target frame, and sets the corrected gaze area as the gaze area for the image of the target frame.

[0008] The video processing program according to the present disclosure causes a computer to perform a prediction process that predicts a gaze area in an image of a target frame based on an object detection result in an image of a frame prior to the target frame, a correction process that corrects the gaze area predicted by the prediction process based on the object detection result and object prediction result in an image of a frame prior to the target frame, and a setting process that sets the corrected gaze area as the gaze area for the image of the target frame.

[0009] According to the present disclosure, it is possible to suppress image quality degradation while performing low-latency video processing.

[0010] 1 is a block diagram showing an example of a configuration of a video processing device. FIG. 1 is an explanatory diagram showing an example of frame storage processing in a detection information storage unit. FIG. 1 is an explanatory diagram showing an example of frame storage processing in the detection information storage unit. FIG. 1 is an explanatory diagram showing an example of an object detection result passed to an ROI prediction unit. FIG. 1 is an explanatory diagram showing an example of an object detection result passed to an ROI correction unit. FIG. 2 is an explanatory diagram showing an example of a prediction of a transition of object coordinates. FIG. 2 is an explanatory diagram showing an example of a calculation formula used for predicting a transition of object coordinates. FIG. 3 is an explanatory diagram showing an example of a prediction of a transition of object size. FIG. 3 is an explanatory diagram showing an example of a calculation formula used for predicting a transition of object size. FIG. 4 is an explanatory diagram showing a case where the object prediction result and the object detection result completely match. FIG. 5 is an explanatory diagram showing a case where the object prediction result and the object detection result partially match (the proportion of overlapping areas is equal to or greater than a threshold). FIG. 6 is an explanatory diagram showing a case where the object prediction result and the object detection result partially match (the proportion of overlapping areas is less than a threshold). FIG. 7 is an explanatory diagram showing a case where the object prediction result and the object detection result do not match at all. A flowchart showing a processing operation of the video processing device. FIG. 8 is a flowchart showing a detailed example of the processes of steps S100 to S300 in the processing operation of the video processing device. Fig. 1 is a flowchart showing a detailed example of the process of step S400 in the processing operation of the video processing device. Fig. 2 is a flowchart showing a detailed example of the process of steps S900 to S1000 in the processing operation of the video processing device. Fig. 3 is a time chart showing an example of the operation of each unit included in the video processing device. Fig. 4 is a schematic block diagram showing the configuration of a computer according to the present disclosure. Fig. 5 is a block diagram showing an overview of the video processing device according to the present disclosure.

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Here, the target for setting an ROI is the image data of the nth frame, and the frames before the nth frame are the n-1th frame, the n-2nd frame, and the n-3rd frame in order of most recent.

[0012] Fig. 1 is a block diagram showing an example of the configuration of a video processing device 100. The video processing device 100 shown in Fig. 1 includes an image input unit 101, a detection information storage unit 102, an ROI prediction unit 103, an ROI correction unit 104, an ROI setting unit 105, an encoding unit 106, and an object detection unit 107.

[0013] The image input unit 101 inputs image data and outputs it to the encoding unit 106 and the object detection unit 107. For example, when the image input unit 101 inputs image data of the nth frame, it outputs the image data of the nth frame to the encoding unit 106 and the object detection unit 107. The image input unit can input images at a predetermined frame rate from outside the video processing device 100, for example.

[0014] The detection information storage unit 102 stores information corresponding to a frame input from the image input unit 101. The detection information storage unit 102 stores, for example, image data of the nth frame and an object detection result for the nth frame in association with each other as information corresponding to the nth frame. In this embodiment, the image input unit 101 inputs image data of the nth frame and outputs it to the object detection unit 107. Next, the object detection unit 107 performs object detection processing on the image data of the nth frame input from the image input unit 101 to obtain an object detection result. Next, the object detection unit 107 stores the obtained object detection result for the nth frame in association with the image data of the nth frame in the detection information storage unit 102. The detection information storage unit 102 memorizes the stored data. The detection information storage unit 102 is realized, for example, by a magnetic disk or the like.

[0015] In this embodiment, an object region detected by an object detection process that detects a specific object included in an image of a target frame is referred to as an object detection result of the target frame. Also, an object region predicted by an object prediction process that predicts a specific object included in an image of a target frame is referred to as an object prediction result of the target frame.

[0016] In addition, in this embodiment, the video processing device 100 can correct the object region predicted as the object prediction result. Therefore, when distinguishing between the object region predicted by the ROI prediction unit 103 and the object region that the ROI setting unit 105 ultimately sets as the ROI for the image of the target frame, the latter is referred to as the ROI prediction result.

[0017] In this embodiment, the object detection result, the object prediction result, and the ROI prediction result each include data specifying at least the center coordinates of the object region and the size of the object region. Hereinafter, the center coordinates of the object region will be referred to as object coordinates, and the size of the object region will be referred to as object size.

[0018] In this embodiment, the detection information storage unit 102 stores image data and object detection results in association with each other as information corresponding to a frame. However, the detection information storage unit 102 may store object detection results instead of image data as information corresponding to a frame. Furthermore, for example, the detection information storage unit 102 may store object detection results and object prediction results as information corresponding to a frame.

[0019] When storing information corresponding to a frame, the detection information storage unit 102 performs different processing depending on the number of pieces of information already stored (i.e., the number of frames). Figures 2 and 3 are explanatory diagrams showing an example of frame storage processing in the detection information storage unit 102.

[0020] As shown in FIG. 2, for example, if the number of pieces of information (i.e., the number of frames) already stored in the detection information storage unit 102 is between 0 and 2 frames, the detection information storage unit 102 stores the information of the newly stored frame as is.

[0021] 3, when the number of frames (i.e., the number of frames) already stored in the detection information storage unit 102 is three, the information corresponding to the oldest frame is deleted and information corresponding to the newly stored frame is stored. In this way, when the detection information storage unit 102 has already stored the maximum number of frames that can be stored, the information corresponding to the oldest frame is deleted and information corresponding to the newest frame is stored.

[0022] The frame information stored in the detection information storage unit 102 is reset when the ROI correction unit 104 determines that the degree of match between the object detection result and the object prediction result (i.e., the overlap ratio of the object regions) is less than a threshold. In this embodiment, the frame information stored in the detection information storage unit 102 is reset by executing the process of step S304 when a negative determination is made in step S302 shown in FIG. 15 . Furthermore, the frame information stored in the detection information storage unit 102 is reset when the object detection unit 107 determines that a scene change has occurred. In this embodiment, the frame information stored in the detection information storage unit 102 is reset by executing the process of step S903 when a positive determination is made in step S902 shown in FIG. 17 . As a method for resetting the frame information stored in the detection information storage unit 102, a method can be applied in which frame information stored before the reset time is not used in the processes performed by the ROI prediction unit 103 and the ROI correction unit 104. Such a method may include, for example, deleting all frame information, or setting a flag so that the information is not referenced by the ROI prediction unit 103 or the ROI correction unit 104 .

[0023] Furthermore, the frame information stored in the detection information storage unit 102 is output to the ROI prediction unit 103 and the ROI correction unit 104 .

[0024] Next, frame information output when an ROI is set for the nth frame will be described. Fig. 4 is an explanatory diagram showing an example of an object detection result transferred to the ROI prediction unit 103. Fig. 5 is an explanatory diagram showing an example of an object detection result transferred to the ROI correction unit 104.

[0025] When an ROI is set for the nth frame, the object detection results for the n-3th, n-2th, and n-1th frames stored in the detection information storage unit 102 are output to the ROI prediction unit 103, as shown in Fig. 4. When an ROI is set for the nth frame, the object detection result for the n-1th frame stored in the detection information storage unit 102 is output to the ROI correction unit 104, as shown in Fig. 5. In this way, even if the frame for which the ROI is set is the same, the frame information output to the ROI prediction unit 103 and the frame information output to the ROI correction unit 104 are different.

[0026] ROI prediction unit 103 calculates the moving speed and moving acceleration of object coordinates and the changing speed and changing acceleration of object size based on the object detection result of the frame output from detection information storage unit 102. Furthermore, ROI prediction unit 103 predicts the ROI (i.e., object region) in a future frame based on the calculated moving speed and moving acceleration of object coordinates and the changing speed and changing acceleration of object size.

[0027] First, the prediction of object coordinates by the ROI prediction unit 103 will be described. FIG. 6 is an explanatory diagram showing an example of the transition prediction of object coordinates. FIG. 7 is an explanatory diagram showing an example of a calculation formula used for the transition prediction of object coordinates. In this embodiment, the frame switching interval is a period T, and the transition occurs in the order of (1), (2), and (3) in FIG. 6 every T periods. Furthermore, each frame in FIG. 6 (1), (2), and (3) is represented by a coordinate system with the lower left point as the origin, X as the horizontal axis, and Y as the vertical axis.

[0028] 6 and 7 show an example in which ROI prediction section 103 predicts object coordinates based on object detection results in the n-3th frame, the n-2th frame, and the n-1th frame.

[0029] As shown in Figures 6(1) and 6(2) and 7(1) and 7(2), the ROI prediction unit 103 calculates the object coordinate displacement during the T period based on the object coordinates in the (n-3)th frame and the object coordinates in the (n-2)th frame. Furthermore, the ROI prediction unit 103 calculates the object movement speed using the object coordinate displacement. Then, the ROI prediction unit 103 predicts the object coordinates in the (n-1)th frame based on the calculated object movement speed and the object coordinates in the (n-2)th frame.

[0030] As shown in FIGS. 6(2) and 6(3) and 7(2) and 7(3), the ROI prediction unit 103 calculates the object coordinate displacement during the T period based on the object coordinates in the n-2th frame and the object coordinates in the n-1th frame. Furthermore, the ROI prediction unit 103 calculates the object movement velocity using the object coordinate displacement. The ROI prediction unit 103 then calculates the object movement acceleration based on the object movement velocity calculated from the n-3th frame and the n-2th frame and the object movement velocity calculated from the n-2th frame and the n-1th frame. The ROI prediction unit 103 then predicts the object coordinates in the nth frame based on the calculated object movement velocity and object movement acceleration and the object coordinates in the n-1th frame. FIG. 6 illustrates an example in which, at the timing of predicting the object coordinates in the n-1th frame shown in FIG. 6(2), the detection information storage unit 102 stores information for two frames as frame information that the ROI prediction unit 103 can use. In this case, the ROI prediction unit 103 can calculate the object movement speed based on information for two frames. However, the ROI prediction unit 103 cannot calculate the object movement acceleration. Therefore, the ROI prediction unit 103 predicts the object coordinates for the (n-1)th frame without using the object movement acceleration. At the timing of predicting the object coordinates for the nth frame shown in FIG. 6 (3), the detection information storage unit 102 stores information for three frames as frame information that the ROI prediction unit 103 can use. In this case, the ROI prediction unit 103 can calculate the object movement acceleration based on information for three frames. Therefore, the ROI prediction unit 103 predicts the object coordinates for the nth frame using the object movement acceleration.

[0031] Next, the prediction of object size by the ROI prediction unit 103 will be described. FIG. 8 is an explanatory diagram showing an example of prediction of transition of object size. FIG. 9 is an explanatory diagram showing an example of a calculation formula used for predicting transition of object size. In the example shown in FIG. 8, the transition occurs in the order of (1), (2), and (3) in FIG. 8 for each period T. Furthermore, the object size in each frame of FIG. 8 (1), (2), and (3) is indicated by W representing the width and H representing the height.

[0032] 8 and 9 show an example in which ROI prediction unit 103 predicts the object size based on the object detection results of the n-3th frame, the n-2th frame, and the n-1th frame.

[0033] As shown in Figures 8(1) and 8(2) and 9(1) and 9(2), the ROI prediction unit 103 calculates the object size displacement during the T period based on the object size in the (n-3)th frame and the object size in the (n-2)th frame. Furthermore, the ROI prediction unit 103 calculates the object size change rate using the object size displacement. Then, the ROI prediction unit 103 predicts the object size in the (n-1)th frame based on the calculated object size change rate and the object size in the (n-2)th frame.

[0034] As shown in FIGS. 8 (2) and (3) and 9 (2) and (3), the ROI prediction unit 103 calculates the object size displacement during the T period based on the object size in the n-2th frame and the object size in the n-1th frame. Furthermore, the ROI prediction unit 103 calculates the object size change velocity using the object size displacement. The ROI prediction unit 103 then calculates the object size change acceleration based on the object size change velocity calculated from the n-3th frame and the n-2th frame and the object size change velocity calculated from the n-2th frame and the n-1th frame. The ROI prediction unit 103 then predicts the object size in the nth frame based on the calculated object size change velocity and object size change acceleration and the object size in the n-1th frame. FIG. 8 illustrates an example in which, at the timing of predicting the object size in the n-1th frame shown in (2), the detection information storage unit 102 stores information for two frames as frame information that the ROI prediction unit 103 can use. In this case, the ROI prediction unit 103 can calculate the object size variation speed based on information for two frames. However, the ROI prediction unit 103 cannot calculate the object size variation acceleration. Therefore, the ROI prediction unit 103 predicts the object size for the (n-1)th frame without using the object size variation acceleration. However, at the timing of predicting the object size for the nth frame shown in (3) in Figure 8, the detection information storage unit 102 stores information for three frames as frame information that the ROI prediction unit 103 can use. In this case, the ROI prediction unit 103 can calculate the object size variation acceleration based on information for three frames. Therefore, the ROI prediction unit 103 predicts the object size for the nth frame using the object size variation acceleration.

[0035] The ROI correction unit 104 compares the object detection result output from the detection information storage unit 102 with the object prediction result predicted by the ROI prediction unit 103. Based on the comparison result, the ROI correction unit 104 corrects the object prediction result for the frame in which the ROI is set.

[0036] When the ROI setting target is the nth frame, the ROI correction unit 104 compares the object detection result for the n-1th frame output from the detection information storage unit 102 with the object prediction result for the n-1th frame predicted by the ROI prediction unit 103. Then, the ROI correction unit 104 corrects the object prediction result for the nth frame based on the comparison result.

[0037] Next, classification of the comparison results between the object prediction result and the object detection result in the ROI correction unit 104 will be described. FIGS. 10 to 13 are explanatory diagrams showing examples of the comparison results between the object prediction result and the object detection result. FIG. 10 is an explanatory diagram showing a case where the object prediction result and the object detection result completely match. FIG. 11 is an explanatory diagram showing a case where the object prediction result and the object detection result partially match (the proportion of overlapping areas is equal to or greater than a threshold). FIG. 12 is an explanatory diagram showing a case where the object prediction result and the object detection result partially match (the proportion of overlapping areas is less than a threshold). FIG. 13 is an explanatory diagram showing a case where the object prediction result and the object detection result do not match at all.

[0038] The results of the comparison between the object detection result and the object prediction result by the ROI correction unit 104 can be classified into, for example, a first comparison result, a second comparison result, and a third comparison result. The first comparison result is a result in which the object prediction result and the object detection result completely match, as shown in Fig. 10. The second comparison result is a result in which the object prediction result and the object detection result partially match, as shown in Figs. 11 and 12. The third comparison result is a result in which the object prediction result and the object detection result do not match at all, as shown in Fig. 13.

[0039] If the comparison result is the first one, the ROI correction unit 104 determines that the object prediction error for the (n-1)th frame is small. The ROI correction unit 104 calculates the difference in object coordinates and the difference in object size between the object prediction result for the (n-1)th frame and the object detection result. The ROI correction unit 104 outputs data indicating the calculated difference as an object region correction result for correcting the object prediction result for the nth frame. The processing by the ROI correction unit 104 corresponds to the processing of steps S303 and S305 shown in FIG. 15 .

[0040] If the comparison result is the third one, the video processing device 100 determines that the object prediction error for the (n-1)th frame is large. The video processing device 100 sets the entire image area of ​​the nth frame as the ROI prediction result. After the comparison result is determined to be the third one, the video processing device 100 stops setting an ROI based on information from past frames and starts setting an ROI based on information from the nth frame onwards.

[0041] For example, the ROI correction unit 104 executes the process of step S304 in FIG. 15 to output a stored information reset signal to the detection information storage unit 102 and reset the frame information stored in the detection information storage unit 102. Then, the ROI prediction unit 103 executes the processes of steps S402 and S409 in FIG. 16 to set the entire region of the image of the nth frame as the object prediction result. By executing these processes, the ROI of the target frame is corrected to the entire region of the image and set. Furthermore, subsequent processes by the ROI prediction unit 103, the ROI correction unit 104, and the ROI setting unit 105 are executed based on the newly stored object detection result.

[0042] If the result is the second comparison result, the ROI correction unit 104 further classifies the result based on the degree of agreement between the object prediction result and the object detection result (i.e., the overlapping ratio of the object region). Specifically, the ROI correction unit 104 determines whether the ratio of the overlapping region between the object prediction result and the object detection result is equal to or greater than an arbitrarily determined threshold. If the ratio is equal to or greater than the threshold, as shown in FIG. 11 , the ROI correction unit 104 calculates and outputs the object region correction result using the same process as in the case of the first comparison result.

[0043] 12, ROI correction unit 104 outputs a memory information reset signal by the same process as in the case of the third comparison result. That is, the ROI of the target frame is corrected and set to the entire area of ​​the image, and the subsequent processes by ROI prediction unit 103, ROI correction unit 104, and ROI setting unit 105 are performed based on the newly stored object detection result.

[0044] ROI setting unit 105 adds the object region correction result for the nth frame obtained from ROI correction unit 104 to the object prediction result for the nth frame obtained from ROI prediction unit 103. ROI setting unit 105 sets the result of the addition as the ROI prediction result for the nth frame. ROI setting unit 105 also outputs the ROI prediction result to encoding unit 106.

[0045] The encoding unit 106 performs encoding processing on the image data of the n-th frame received from the image input unit 101, using the ROI prediction result set by the ROI setting unit 105 as the ROI.

[0046] The object detection unit 107 performs object detection processing on the image data of the nth frame received from the image input unit 101, and generates information specifying the object coordinates and object size as the object detection result. The object detection unit 107 also stores the generated object detection result in the detection information storage unit 102 as information for a predetermined frame. The detection information storage unit 102 memorizes the stored data. In this embodiment, when performing object detection processing, when the object is regarded as a rectangle, the coordinates of the center of the rectangle are set as the center coordinates of the object area, and the length (height) x width (width) of the rectangle is set as the size of the object area. However, the method of determining the object area is not limited to this.

[0047] The image input unit 101, the ROI prediction unit 103, the ROI correction unit 104, the ROI setting unit 105, the encoding unit 106, and the object detection unit 107 are realized by a CPU of a computer that operates according to a program (video processing program). For example, the program may be stored in a storage unit of the video processing device 100, and the CPU may read the program and operate as the image input unit 101, the ROI prediction unit 103, the ROI correction unit 104, the ROI setting unit 105, the encoding unit 106, and the object detection unit 107 according to the program. Alternatively, the image input unit 101, the ROI prediction unit 103, the ROI correction unit 104, the ROI setting unit 105, the encoding unit 106, and the object detection unit 107 may each be realized by dedicated hardware.

[0048] Next, the operation of the video processing device 100 when the ROI is set to the image data of the nth frame will be described. Fig. 14 is a flowchart showing the processing operation of the video processing device.

[0049] The ROI correction unit 104 refers to the object prediction result of the (n-1)th frame (step S100).

[0050] The object prediction result for the n-1th frame referenced in step S100 may be obtained, for example, by object prediction processing for the n-1th frame by ROI prediction unit 103 before step S100. Also, for example, when the ROI setting target is the n-1th frame, the object prediction result obtained by object prediction processing for the n-1th frame by ROI prediction unit 103 (or the object prediction result reflecting the object region correction result by ROI correction unit 104) may be stored in a storage unit such as detection information storage unit 102, and when the ROI setting target is the nth frame, the stored object prediction result for the n-1th frame (or the object prediction result reflecting the object region correction result) may be referenced.

[0051] Next, the ROI correction unit 104 refers to the object detection result of the (n-1)th frame stored in the detection information storage unit 102 (step S200).

[0052] Next, the ROI correction unit 104 compares the object coordinates and object size using the object detection result and the object prediction result of the (n-1)th frame, and outputs the object region correction result to the ROI setting unit 105 based on the comparison result (step S300).

[0053] Next, a detailed example of the processing of steps S100 to S300 will be described below. Fig. 15 is a flowchart showing a detailed example of the processing of steps S100 to S300 in the processing operation of the video processing device.

[0054] As shown in FIG. 15, after the process of step S200 is completed, the ROI correction unit 104 compares the object coordinates and the object size using the object detection result and the object prediction result of the (n-1)th frame (step S301).

[0055] Next, the ROI correction unit 104 determines whether the degree of match between the object detection result and the object prediction result in the (n-1)th frame (ie, the overlap rate of the object regions) is equal to or greater than a threshold (step S302).

[0056] If it is determined that the degree of coincidence between the object prediction result and the object detection result (i.e., the overlap rate of the object region) is equal to or greater than a threshold, i.e., if it is determined that the error in the object prediction is small, the ROI correction unit 104 calculates the difference in object coordinates and object size using the object detection result and the object prediction result for the (n-1)th frame (step S303).Then, the ROI correction unit 104 outputs the calculation result to the ROI setting unit 105 as the object region correction result for the nth frame (step S305).

[0057] If it is determined that the degree of agreement between the object prediction result and the object detection result (i.e., the overlap ratio of the object regions) is less than the threshold, i.e., if it is determined that the error in the object prediction is large, the ROI correction unit 104 outputs a memory information reset signal to the detection information storage unit 102 (step S304). The ROI correction unit 104 outputs the object region correction result to the ROI setting unit 105 (step S305). If step S305 is executed after step S304, the ROI correction unit 104 may output data indicating that the entire region of the image is set as the object region correction result. By outputting the memory information reset signal in step S304, the entire region of the image is set as the object prediction result in the processing of step S409 shown in FIG. 16. Therefore, for example, the ROI correction unit 104 may output data indicating no correction as the object region correction result in step S305. This completes the processing of step S300.

[0058] 14, the ROI prediction unit 103 performs object prediction processing for the n-th frame based on frames prior to the n-th frame. The ROI prediction unit 103 outputs the object prediction result to the ROI setting unit 105.

[0059] Next, a detailed example of the process of step S400 will be described below. Fig. 16 is a flowchart showing a detailed example of the process of step S400 in the processing operation of the video processing device.

[0060] 16 , the ROI prediction unit 103 receives object detection results for all frames stored in the detection information storage unit 102 (step S401). When the ROI setting target is image data for the nth frame, the object detection results for the nth frame have not yet been stored in the detection information storage unit 102. Therefore, the ROI prediction unit 103 receives object detection results for frames prior to the nth frame.

[0061] Next, the ROI prediction unit 103 determines whether the received data is equal to or less than one frame (step S402).

[0062] If it is determined that the total area is equal to or less than one frame, the ROI prediction unit 103 determines an object prediction result in which the entire image area is the object area (step S409).The ROI prediction unit 103 outputs the determined object prediction result to the ROI setting unit 105 (step S408).

[0063] On the other hand, if it is determined that the length is not equal to or less than one frame, the ROI prediction unit 103 calculates the object movement speed and the object size change speed based on the received data (step S403).

[0064] As described above, when ROI prediction unit 103 receives object detection results for the n-3th frame, the n-2th frame, and the n-1th frame, it calculates the object movement speed and the object size change speed between the n-3th frame and the n-2th frame based on the object detection results for the n-3th frame and the n-2th frame. Also, ROI prediction unit 103 calculates the object movement speed and the object size change speed between the n-2th frame and the n-1th frame based on the object detection results for the n-2th frame and the n-1th frame.

[0065] Also, for example, when the ROI prediction unit 103 receives the object detection results for the n-2th frame and the n-1th frame, it calculates the object movement speed and object size change speed between the n-2th frame and the n-1th frame based on the object detection results for the n-2th frame and the n-1th frame.

[0066] Next, the ROI prediction unit 103 determines whether the received data is for two frames (step S402).

[0067] If it is determined that the received data does not correspond to two frames, i.e., if three frames of object detection results have been received, the ROI prediction unit 103 calculates the object movement acceleration and the object size change acceleration based on the received data (step S405).Then, the process proceeds to step S407.

[0068] For example, when the ROI prediction unit 103 receives object detection results for the n-3 frame, the n-2 frame, and the n-1 frame, it calculates the object movement acceleration and object size change acceleration between the n-2 frame and the n-1 frame based on the object movement speed and object size change speed between the n-3 frame and the n-2 frame calculated in step S403, and the object movement speed and object size change speed between the n-2 frame and the n-1 frame.

[0069] On the other hand, if it is determined that the received data is for two frames, i.e., if object detection results for two frames, the (n-2)th frame and the (n-1)th frame, are received, ROI prediction unit 103 sets the object movement acceleration and object size change acceleration between the (n-2)th frame and the (n-1)th frame to 0 (step S403).Then, the process proceeds to step S407.

[0070] In step S407, the ROI prediction unit 103 predicts the object coordinates for the nth frame based on the object coordinates, object movement speed, and object movement acceleration at the (n-1)th frame. The ROI prediction unit 103 also predicts the object size for the nth frame based on the object size, object size change speed, and object size change acceleration at the (n-1)th frame.

[0071] Next, ROI prediction unit 103 outputs the object region specified by the object coordinates and object size predicted in step S407, or the entire region set in step S409, as an object prediction result to ROI setting unit 105 (step S408). This completes the processing of step S400.

[0072] 14, the ROI setting unit 105 applies the object region correction result output from the ROI correction unit 104 to the object prediction result output from the ROI prediction unit 103 (step S500). The ROI setting unit 105 sets the result of application as the ROI prediction result for the nth frame (step S600). The ROI setting unit 105 outputs the ROI prediction result to the encoding unit 106.

[0073] The encoding unit 106 receives image data of the nth frame input from the image input unit 101 (step S700). The encoding unit 106 performs encoding processing on the image data of the nth frame using the ROI prediction result set by the ROI setting unit 105 as the ROI (step S800).

[0074] The object detection unit 107 performs object detection processing on the image data of the nth frame input from the image input unit 101, and acquires data indicating object coordinates and object size as the object detection result (step S900). The object detection unit 107 stores the acquired object detection result of the nth frame in the detection information storage unit 102. The detection information storage unit 102 stores the object detection result (step S1000).

[0075] Next, a detailed example of the processing of steps S900 to S1000 will be described below. Fig. 17 is a flowchart showing a detailed example of the processing of steps S900 to S1000 in the processing operation of the video processing device.

[0076] 17 , the object detection unit 107 receives image data of the nth frame input from the image input unit 101 (step S901). The object detection unit 107 determines whether a scene change has occurred based on the image data (step S902). For example, the object detection unit 107 detects the similarity between the image of the nth frame and the image of the (n−1)th frame, and determines that a scene change has occurred when the detected similarity is less than an arbitrary threshold.

[0077] If it is determined that a scene change has not occurred, the process proceeds to step S904. If it is determined that a scene change has occurred, the object detection unit 107 outputs a stored information reset signal that resets information corresponding to frames prior to the nth frame stored in the detection information storage unit 102 (step S903). Thereafter, the process proceeds to step S904. By executing the process of step S903, the information corresponding to frames prior to the nth frame stored in the detection information storage unit 102 is no longer referenced by the ROI prediction unit 103 or the ROI correction unit 104.

[0078] Next, the object detection unit 107 performs an object detection process on the image of the nth frame (step S904). The object detection unit 107 outputs data indicating the object coordinates and object size obtained by performing the object detection process as the object detection result for the nth frame to the detection information storage unit 102 in association with the image data of the nth frame (step S905).

[0079] If the number of stored frames is less than the upper limit (the upper limit is 3 in this embodiment) (Yes in step S1001), the detection information storage unit 102 stores the image data of the nth frame and the object detection result output from the object detection unit 107 in a free area (step S1002), and then completes storing the information corresponding to the nth frame (step S1004).

[0080] If the number of stored frames has reached the upper limit (No in step S1001), the detection information storage unit 102 updates the information corresponding to the oldest frame among the stored information to the image data and object detection result of the nth frame output from the object detection unit 107 (step S1003), and then completes storing the information corresponding to the nth frame (step S1004).

[0081] Next, a description will be given with reference to a time chart of the operation of each unit included in the video processing device 100. Fig. 18 is a time chart showing an example of the operation of each unit included in the video processing device.

[0082] In the example shown in Figure 18, image input unit 101 starts inputting image data for the (n-2)th frame at timing T1. Image input unit 101 starts inputting image data for the (n-1)th frame at timing T2, a period T later. Image input unit 101 starts inputting image data for the nth frame at timing T3, a further period T later. Image input unit 101 starts inputting image data for the (n+1)th frame at timing T4, a further period T later. Image input unit 101 starts inputting image data for the (n+2)th frame at timing T5, a further period T later.

[0083] As shown in Figure 18, at timing T3 when input of image data for the nth frame begins, the object detection results for the n-3th frame, the n-2th frame, and the n-1th frame are stored in the detection information storage unit 102.

[0084] 18 , at timing T3, the ROI prediction unit 103 receives the object detection results for the (n-3), (n-2), and (n-1) frames. Based on these, the ROI prediction unit 103 executes object prediction processing for the nth frame. The ROI prediction unit 103 outputs the object prediction result for the nth frame to the ROI setting unit 105 (corresponding to the processing in step S400).

[0085] 18, at timing T3, the ROI correction unit 104 receives the object detection result and the object prediction result for the (n-1)th frame (corresponding to the processing of steps S100 to S200).The ROI correction unit 104 outputs the comparison result to the ROI setting unit 105 as the object prediction correction result for the nth frame (corresponding to the processing of step S300).

[0086] Then, as shown in FIG. 18, after the object prediction result for the nth frame is output from the ROI prediction unit 103 to the ROI setting unit 105 and the object prediction correction result for the nth frame is output from the ROI correction unit 104 to the ROI setting unit 105, the ROI setting unit 105 outputs the ROI prediction result obtained by applying the object region correction result to the object prediction result to the encoding unit 106 (corresponding to the processing of steps S500 to S600).

[0087] 18, after the ROI prediction result is output from ROI setting unit 105 to encoding unit 106, encoding unit 106 performs encoding processing using the ROI prediction result as the ROI for the image data of the nth frame input from image input unit 101. Furthermore, encoding unit 106 outputs the encoded data (corresponding to the processing of steps S700 to S800).

[0088] 18 , after the encoding unit 106 starts encoding the nth frame, the detection information storage unit 102 erases the object detection result for the (n-3)th frame and stores a new object detection result for the nth frame by the object detection unit 107 (corresponding to the processing of steps S900 to S1000).

[0089] In this embodiment, the ROI prediction unit 103 predicts the ROI based on information obtained from the object detection result of a frame prior to the frame for which the ROI is to be set. Furthermore, the ROI correction unit 104 compares the object prediction result of the frame prior to the frame for which the ROI is to be set with the object detection result. The ROI correction unit 104 corrects the predicted ROI based on the comparison result. With this configuration, when setting an ROI in a target frame and performing encoding processing, there is no need to wait for the frame to be read before performing the object detection processing, thereby reducing overall processing delay. Furthermore, correcting the predicted ROI can reduce image quality degradation. Therefore, image quality degradation can be reduced while achieving low-latency video processing.

[0090] In addition, in this embodiment, the object prediction results and object detection results of frames prior to the frame for which the ROI is to be set are compared. If it is determined that the degree of match between the object prediction results and the object detection results (i.e., the overlap rate of the object regions) is less than a threshold, i.e., if it is determined that the error in the object prediction is large, the object detection results of past frames are reset. The ROI prediction unit 103 predicts the ROI based on the object detection results obtained after the reset. With this configuration, if the predicted ROI does not match the ROI to be set, the past information used for prediction is reset. As a result, continuous degradation of image quality can be suppressed.

[0091] 19 is a schematic block diagram showing the configuration of a computer according to the present disclosure. The computer 1000 includes a processor 1001 such as a CPU (Central Processing Unit), a program memory 1002, a storage medium 1003 for storing video data, and a storage medium 1004 for storing a bitstream. The storage medium 1003 and the storage medium 1004 may be separate storage media or may be storage areas of the same storage medium. A magnetic storage medium such as a hard disk can be used as the storage medium.

[0092] The program memory 1002 stores a program (video processing program) for implementing the functions of each block (excluding the buffer block) described in the above embodiment. The processor 1001 then executes processing in accordance with the program stored in the program memory 1002, thereby implementing the functions of the video processing device 100 described in the above embodiment. That is, the processor 1001 implements the functions of the image input unit 101, ROI prediction unit 103, ROI correction unit 104, ROI setting unit 105, encoding unit 106, and object detection unit 107 shown in FIG. 1 .

[0093] At least the program memory 1002 is a non-transitory computer-readable medium. However, the program may be stored in various types of transitory computer-readable medium. The program is supplied to the transitory computer-readable medium via, for example, a wired or wireless communication path, i.e., via an electrical signal, an optical signal, or an electromagnetic wave.

[0094] Next, an overview of the present disclosure will be described. Fig. 20 is a block diagram illustrating an overview of a video processing device according to the present disclosure. The video processing device 10 shown in Fig. 20 includes: prediction means 11 (implemented by an ROI prediction unit 103 in the embodiment) that predicts a gaze area (in the embodiment, an ROI) in an image of a target frame based on an object detection result in an image of a frame prior to the target frame; correction means 12 (implemented by an ROI correction unit 104, an ROI prediction unit 103, or an ROI setting unit 105 in the embodiment) that corrects the gaze area predicted by the prediction means 11 based on the object detection result and the object prediction result in the image of the frame prior to the target frame; and setting means 13 (implemented by the ROI setting unit 105 in the embodiment) that sets the corrected gaze area as the gaze area for the image of the target frame.

[0095] With this configuration, when setting a gaze area in a target frame and performing encoding processing, there is no need to wait for the frame to be read before performing object detection processing, thereby reducing delays in the entire processing. Furthermore, correcting the predicted gaze area can reduce image quality degradation. Therefore, it is possible to achieve low-latency video processing while suppressing image quality degradation.

[0096] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0097] (Appendix 1) A video processing device that sets a gaze area and performs video processing, comprising: a prediction means that predicts the gaze area in an image of a target frame based on an object detection result in an image of a frame prior to the target frame; a correction means that corrects the gaze area predicted by the prediction means based on the object detection result and object prediction result in an image of a frame prior to the target frame; and a setting means that sets the corrected gaze area as the gaze area for the image of the target frame.

[0098] (Appendix 2) The prediction means of the video processing device described in Appendix 1 predicts the gaze area based on the object detection results of frames after the target frame when the degree of match between the object detection results and the object prediction results in images of frames prior to the target frame is less than a predetermined threshold.

[0099] (Appendix 3) An image processing device as described in Appendix 2, comprising a storage means for storing object detection results for each frame, wherein the prediction means predicts the gaze area in the image of the target frame based on the object detection results of the frame stored in the storage means, and the object detection result of the frame stored in the storage means is reset if the degree of match between the object detection result in the image of the frame prior to the target frame and the object prediction result is less than a predetermined threshold.

[0100] (Appendix 4) A video processing device as described in Appendix 1 or Appendix 2, wherein the correction means corrects the gaze area to the entire area in the image of the target frame when the degree of match between the object detection result and the object prediction result in an image of a frame prior to the target frame is less than a predetermined threshold.

[0101] (Appendix 5) The video processing device described in Appendix 4, wherein the correction means, when the degree of match between the object detection result and the object prediction result in an image of a frame prior to the target frame exceeds a predetermined threshold, applies the difference between the object detection result and the object prediction result to the gaze area predicted by the prediction means to correct the gaze area.

[0102] (Supplementary Note 6) The video processing device according to Supplementary Note 1 or Supplementary Note 2, comprising an encoding means for performing encoding processing on the image of the target frame based on the gaze area set by the setting means.

[0103] (Appendix 7) A video processing method applied to a computer that sets a gaze area and processes video, characterized in that the computer predicts a gaze area in an image of a target frame based on an object detection result in an image of a frame earlier than the target frame, corrects the predicted gaze area based on the object detection result and object prediction result in an image of a frame earlier than the target frame, and sets the corrected gaze area as the gaze area for the image of the target frame.

[0104] (Appendix 8) A video processing method as described in Appendix 7, in which the computer predicts the gaze area based on the object detection results of frames after the target frame if the degree of match between the object detection results and the object prediction results in an image of a frame prior to the target frame is less than a predetermined threshold.

[0105] (Appendix 9) A video processing method as described in Appendix 8, in which the computer predicts the gaze area in the image of the target frame based on the object detection results of the frame stored in the memory unit for each frame, and if the degree of match between the object detection result in the image of the frame prior to the target frame and the object prediction result is less than a predetermined threshold, resets the object detection result of the frame stored in the memory unit.

[0106] (Appendix 10) A video processing method as described in Appendix 7 or Appendix 8, in which the computer corrects the gaze area to the entire area of ​​the image of the target frame if the degree of match between the object detection result and the object prediction result in an image of a frame prior to the target frame is less than a predetermined threshold.

[0107] (Appendix 11) A video processing method as described in Appendix 10, in which, when the degree of match between the object detection result and the object prediction result in an image of a frame prior to the target frame exceeds a predetermined threshold, the computer applies the difference between the object detection result and the object prediction result to a predicted gaze area to correct the gaze area.

[0108] (Supplementary Note 12) The video processing device according to Supplementary Note 7 or Supplementary Note 8, wherein the computer performs encoding processing on the image of the target frame based on a set gaze area.

[0109] (Appendix 13) A video processing program applied to a computer that sets a gaze area and processes video, the video processing program causing the computer to execute a prediction process that predicts a gaze area in an image of a target frame based on an object detection result in an image of a frame earlier than the target frame, a correction process that corrects the gaze area predicted by the prediction process based on the object detection result and object prediction result in an image of a frame earlier than the target frame, and a setting process that sets the corrected gaze area as the gaze area for the image of the target frame.

[0110] (Appendix 14) A video processing program as described in Appendix 13 for causing a computer to predict a gaze area based on object detection results in frames after the target frame when, in the prediction process, the degree of match between object detection results in an image of a frame prior to the target frame and object prediction results is less than a predetermined threshold.

[0111] (Appendix 15) A video processing program as described in Appendix 14 for causing a computer to execute a process in which, in the prediction process, a gaze area in an image of the target frame is predicted based on the object detection results of the frame stored by a memory unit for each frame, and if the degree of match between the object detection results in an image of a frame prior to the target frame and the object prediction results is less than a predetermined threshold, the object detection results of the frame stored in the memory unit are reset.

[0112] (Appendix 16) A video processing program as described in Appendix 13 or Appendix 14 for causing a computer to correct the gaze area to the entire area in the image of the target frame when, in the correction process, the degree of match between the object detection result and the object prediction result in an image of a frame prior to the target frame is less than a predetermined threshold.

[0113] (Appendix 17) A video processing program as described in Appendix 16, which causes a computer to correct a gaze area predicted by the prediction process by applying the difference between the object detection result and the object prediction result to the gaze area predicted by the prediction process when the degree of match between the object detection result and the object prediction result in an image of a frame prior to the target frame exceeds a predetermined threshold during the correction process.

[0114] (Supplementary Note 18) A video processing program according to Supplementary Note 13 or Supplementary Note 14, which causes a computer to perform an encoding process on an image of the target frame based on the attention area set in the setting process.

[0115] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0116] 10, 100 Video processing device 11 Prediction means 12 Correction means 13 Setting means 101 Image input unit 102 Detection information storage unit 103 ROI prediction unit 104 ROI correction unit 105 ROI setting unit 106 Encoding unit 107 Object detection unit 1000 Computer 1001 Processor 1002 Program memory 1003, 1004 Storage medium

Claims

1. A video processing device which sets a gaze area and performs video processing, comprising: a prediction means for predicting the gaze area in an image of a target frame based on an object detection result in an image of a frame prior to the target frame; a correction means for correcting the gaze area predicted by the prediction means based on the object detection result and the object prediction result in an image of a frame prior to the target frame; and a setting means for setting the corrected gaze area as the gaze area for the image of the target frame.

2. The image processing device according to claim 1, wherein the prediction means predicts the gaze area based on the object detection results in frames after the target frame when the degree of match between the object detection results in an image of a frame prior to the target frame and the object prediction results is less than a predetermined threshold.

3. An image processing device as described in claim 2, further comprising a memory means for storing object detection results for each frame, wherein the prediction means predicts the gaze area in the image of the target frame based on the object detection results of the frames stored in the memory means, and the object detection result of the frame stored in the memory means is reset if the degree of match between the object detection result and the object prediction result in an image of a frame prior to the target frame is less than a predetermined threshold.

4. A video processing device as described in claim 1 or claim 2, wherein the correction means corrects the gaze area to the entire area of ​​the image of the target frame when the degree of match between the object detection result and the object prediction result in an image of a frame prior to the target frame is less than a predetermined threshold.

5. The image processing device of claim 4, wherein the correction means, when the degree of match between the object detection result and the object prediction result in an image of a frame prior to the target frame exceeds a predetermined threshold, applies the difference between the object detection result and the object prediction result to the gaze area predicted by the prediction means to correct the gaze area.

6. A video processing device according to claim 1 or claim 2, further comprising: encoding means for performing encoding processing on the image of the target frame based on the attention area set by said setting means.

7. A video processing method applicable to a computer that sets a gaze area and processes video, the video processing method comprising the steps of: predicting a gaze area in an image of a target frame based on an object detection result in an image of a frame prior to the target frame; correcting the predicted gaze area based on the object detection result and the object prediction result in the image of the frame prior to the target frame; and setting the corrected gaze area as the gaze area for the image of the target frame.

8. A computer-readable recording medium having recorded thereon a video processing program applicable to a computer that sets a gaze area and processes video, the video processing program causing the computer to execute the following processes: a prediction process that predicts a gaze area in an image of a target frame based on an object detection result in an image of a frame prior to the target frame; a correction process that corrects the gaze area predicted by the prediction process based on the object detection result and the object prediction result in an image of a frame prior to the target frame; and a setting process that sets the corrected gaze area as the gaze area for the image of the target frame.

Citation Information

Patent Citations

  • Picture communication system

    JP1994319134A

  • Image processing device and method

    WO2020004073A1

  • Target tracking method, target tracking system, and target tracking program

    WO2020080535A1