Image encoding device, image encoding method, and program

Abstract

This image encoding device comprises: an image encoding unit that performs compression encoding for compressing a plurality of input images, which are included in video set to a processing resolution, by using a lossy encoding method; a quantization parameter derivation unit that derives an average quantization parameter per frame for each prescribed number of frames; and a resolution determination unit that determines the processing resolution by comparing the value of the average quantization parameter with at least one predetermined threshold value for each resolution.

Description

Image encoding device, image encoding method, and program

[0001] The disclosed technology relates to an image encoding device, an image encoding method, and a program.

[0002] There is a growing need for AI-based video analysis, and it is expected to be applied to various fields such as autonomous driving and surveillance. Generally, advanced AI-based video analysis requires a large amount of calculation, so attention is being paid to edge-cloud collaborative systems that achieve advanced processing by linking the cloud, which has fewer resource constraints and excellent scalability, with edge devices, which excel in real-time performance.

[0003] Non-Patent Document 1 proposes an event-driven inference system that detects events by performing AI inference processing with a relatively low computational load on the edge terminal, and the edge terminal transmits video to the cloud when the event is detected, where more advanced AI inference processing is performed. This makes it possible to further reduce unnecessary data transmission between the edge and the cloud and resource consumption in the cloud.

[0004] On the other hand, even in the event-driven inference system described above, if frequent events are detected on many edge devices, the transmission cost of transmitting video to the cloud can become an issue. Therefore, it is desirable for the edge device to compress the video as much as possible before transmitting it, taking into account the decrease in inference accuracy in the cloud.

[0005] As a method for reducing the video transmission bandwidth while maintaining the inference accuracy on the cloud side as much as possible, the technology described in Non-Patent Document 2 has been proposed. The technology described in Non-Patent Document 2 combines Versatile Video Coding (VVC), a video encoding method, with an image reduction / enlargement filter, reducing the resolution of the video to half of the original before encoding it with VVC, and then enlarging the decoded video to the original resolution.

[0006] Eda, et al., “Highly Efficient Event-Driven Inference for AI Services in the IOWN Era,” NTT Technical Journal, 2020; Shindo, et al., “Video Coding Method for YOLO-v7 Combining VVC and CNN,” IEICE General Conference, 2023

[0007] Although the technology described in Non-Patent Document 2 suppresses the deterioration of object detection accuracy when transmitting video at a low bit rate, there is still room for improvement in terms of the deterioration of video quality.

[0008] The technology disclosed herein has been developed in consideration of the above points, and provides an image encoding device, an image encoding method, and a program that can suppress a decline in inference accuracy while also suppressing a decline in video quality more than conventional technologies.

[0009] The image encoding device of the present disclosure includes an image encoding unit that performs compression encoding by compressing multiple input images included in a video having a processing resolution using a lossy encoding method, a quantization parameter derivation unit that derives an average quantization parameter per frame for each predetermined number of frames, and a resolution determination unit that determines the processing resolution by comparing the value of the average quantization parameter with at least one threshold value predetermined for each resolution.

[0010] In addition, the image encoding method disclosed herein performs compression encoding by compressing multiple input images included in a video having a processing resolution using a lossy encoding method, deriving an average quantization parameter per frame for each predetermined number of frames, and determining the processing resolution by comparing the value of the average quantization parameter with at least one threshold value predetermined for each resolution.

[0011] Furthermore, the program of the present disclosure causes a computer to function as a quantization parameter derivation unit and a resolution determination unit of the image encoding device of the present disclosure.

[0012] The technology disclosed herein has been made in consideration of the above points, and can suppress a decrease in inference accuracy while also suppressing a decrease in video quality more than conventional technologies.

[0013] 1 is a configuration diagram showing an example of the configuration of an event-driven inference system of an embodiment. FIG. 1 is a configuration diagram showing an example of the configuration of a video transmission module of an embodiment. FIG. 2 is a block diagram showing an example of the configuration of an image encoding device of an embodiment. FIG. 3 is a diagram for explaining the flow of processing in the image encoding device of an embodiment. FIG. 4 is a graph for explaining the relationship between a quantization parameter, a bit rate, and inference accuracy used during compression encoding in the image encoding unit of an embodiment. FIG. 5 is a diagram for explaining an example in which a control unit of an embodiment switches resolution on a sequence-by-sequence basis. FIG. 6 is a diagram for explaining an example in which a control unit of an embodiment switches resolution on a frame-by-frame basis. FIG. 7 is a flowchart showing an example of the flow of processing resolution control processing executed in a control unit of an embodiment. FIG. 8 is a diagram for explaining a change in an evaluation index when the processing resolution setting is switched from a reduction OFF state to a reduction ON state. FIG. 9 is a diagram for explaining a change in an evaluation index when the processing resolution setting is switched from a reduction ON state to a reduction OFF state. FIG. 10 is a flowchart showing an example of the flow of processing resolution control processing executed in a control unit of a modified example.

[0014] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals.

[0015] First, an example of the configuration of an event-driven inference system according to this embodiment will be described. Fig. 1 shows an example of an event-driven inference system 10 according to this embodiment.

[0016] The event-driven inference system 10 shown in FIG. 1 includes a cloud 12 and an edge terminal group 15. The edge terminal group 15 includes a plurality of edge terminals 14, and in the present embodiment, as an example shown in FIG. 1 , includes a surveillance camera 14A, a drone 14B, an automobile 14C, and a mobile terminal 14D. Note that the number and types of edge terminals 14 included in the edge terminal group 15 are not limited to this embodiment. Note that in the present embodiment, the surveillance camera 14A, the drone 14B, the automobile 14C, and the mobile terminal 14D included in the edge terminal group 15 are collectively referred to as "edge terminals 14" when not distinguishing between them individually.

[0017] Each edge terminal 14 is wirelessly connected to the cloud 12 via a network (not shown). The edge terminal 14 detects a predetermined event by performing a predetermined inference process, and transmits video to the cloud 12 at the timing of detecting the event. Note that in this embodiment, "video" refers to a moving image including multiple images (frames).

[0018] The cloud 12 is a so-called cloud computing system, and is constructed in a data center or the like. The cloud 12 performs more advanced inference processing on the video obtained from the edge terminal 14. The inference results obtained by the inference processing in the cloud 12 can be used in other service applications 4. The cloud 12 can also control the operation of the edge terminal 14 by sending instructions to the edge terminal 14 according to the obtained inference results.

[0019] The edge terminal 14 of this embodiment performs predetermined inference processing on video as described above, and includes a video transmission module 20, an example of which is shown in Fig. 2. The video transmission module 20 shown in Fig. 2 includes a camera 22, an image encoding device 24, a memory 26, and a wireless module 28.

[0020] The camera 22 is an example of a device for acquiring video in the edge terminal 14. The camera 22 outputs the video obtained by capturing the video to the image encoding device 24. Note that, in this embodiment, a form in which the image encoding device 24 processes the video obtained by the camera 22 will be described, but the image to be processed by the image encoding device 24 is not limited to the video obtained by the camera 22. For example, the image may be a moving image such as an animation composed of multiple images drawn by a user in the edge terminal 14. In this embodiment, the video input from the camera 22 to the image encoding device 24 is an example of the multiple input images of the present disclosure.

[0021] The image encoding device 24 is a device, such as an accelerator, that performs compression encoding and the like on the video input from the camera 22. The image encoding device 24 outputs a bit stream obtained by the compression encoding to the memory 26. The wireless module 28 reads the bit stream stored in the memory 26 and transmits it to the cloud 12 when an event is detected, based on instructions from a control unit 34 (see FIG. 3 ) provided in the image encoding device 24.

[0022] The memory 26 stores a bitstream obtained by the image encoding device 24, etc. The memory 26 also stores a resolution control processing program 27 executed by the image encoding device 24. The resolution control processing program 27 may be a single program or a group of programs configured by multiple programs or modules. The memory 26 is configured by an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The resolution control processing program 27 of this embodiment is an example of a program disclosed herein.

[0023] Next, the image encoding device 24 of this embodiment will be described in detail. First, the configuration of the image encoding device 24 will be described. Fig. 3 shows a block diagram illustrating an example of the hardware configuration of the image encoding device 24. Fig. 4 shows a block diagram illustrating the processing flow in the image encoding device 24.

[0024] 3, the image encoding device 24 of this embodiment includes a preprocessing unit 30, an image encoding unit 32, a control unit 34, a memory IF (Interface) 38, an image processing unit 40, and an external IF 42. Each component is connected to each other so as to be able to communicate with each other via a bus 49 such as a system bus or a control bus.

[0025] The pre-processing unit 30 converts the video input from the camera 22 into a format that can be processed by the image encoding unit 32, and writes the converted video to the memory 26 via the memory IF 38. As an example, the pre-processing unit 30 of this embodiment converts each image included in the video input from the camera 22 into the YUV420 format.

[0026] The image processing unit 40 reads from the memory 26 the image whose format has been converted to YUV420 by the preprocessing unit 30, converts it to the processing resolution set by the control unit 34, and then writes the image whose resolution has been converted to the processing resolution into the memory. Note that in this embodiment, the image processing unit 40 is provided separately from the preprocessing unit 30 and the image encoding unit 32, but the preprocessing unit 30 and the image encoding unit 32 may have functions related to the resolution conversion performed by the image processing unit 40. For example, the amount of access to the memory 26 can be further reduced by having the image encoding unit 32 read the image written to the memory 26 by the preprocessing unit 30 while converting the resolution.

[0027] The image encoding unit 32 performs compression encoding on the input video, compressing an image set to a processing resolution using a lossy encoding method. The image encoding unit 32 of this embodiment is a so-called encoder. The image encoding unit 32 reads the video written to the memory 26, performs compression encoding in accordance with a moving image coding standard such as H.264 / AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding), and writes the resulting bitstream to the memory 26. The image encoding unit 32 also performs encoding processing in accordance with the processing resolution determined by the control unit 34.

[0028] The control unit 34 has a function of determining the resolution (processing resolution) of the image when compressed by the image encoding unit 32. In this embodiment, a CPU (Central Processing Unit) is used as the control unit 34. The control unit 34 in this embodiment executes a resolution control processing program 27 stored in the memory 26, so that the CPU functions as the quantization parameter derivation unit 34A and the resolution determination unit 34B shown in FIG.

[0029] The quantization parameter derivation unit 34A derives an average quantization parameter (QP) per frame for each predetermined number of frames.

[0030] The resolution determination unit 34B determines the processing resolution by comparing the average quantization parameter value derived by the quantization parameter derivation unit 34A with at least one threshold value predetermined for each resolution.

[0031] An example of a method for the resolution determination unit 34B to determine the processing resolution will be described with reference to Fig. 5. Fig. 5 shows a graph representing the relationship between the quantization parameter QP used during compression encoding in the image encoding unit 32, the bit rate, and the inference accuracy. Note that in this embodiment, mAP (Mean Average Precision), a general evaluation index for object detection accuracy, is used as the inference accuracy.

[0032] The graph shown in Figure 5 plots the bit rate obtained when all frames (images) of a specified video are compression-encoded using a fixed quantization parameter QP, and the evaluation index mAP for the decoded video. The solid line in the graph shown in Figure 5 indicates the case where the video is compression-encoded at the original resolution, and the dotted line indicates the case where the video is reduced (compressed) at half the original resolution (reduction rate = 1 / 2). In addition, in Figure 5, the quantization parameter QP when compression-encoded at the original resolution is plotted on the graph as "QPo", and the quantization parameter QP when compression-encoded at half the original resolution is plotted as "QPs" so that they can be distinguished.

[0033] In the graph shown in (a), at the original resolution, the evaluation index mAP drops sharply, with the quantization parameter QP changing from approximately 37 to 42. On the other hand, at a resolution half the original, the evaluation index mAP drops sharply, with the quantization parameter QP changing from approximately 42 to 47.

[0034] The graph shown in (b) represents the relationship between the quantization parameter QP, bit rate, and inference accuracy for video that is relatively less difficult to compress and encode than the graph shown in (a). Examples of video that are relatively less difficult to compress and encode include video with many flat shapes and few edges and noise. The graph shown in (b) shows a tendency for the evaluation index mAP to remain constant even at lower bit rates than the graph shown in (a). Meanwhile, the quantization parameter QP corresponding to the change point at which the evaluation index mAP begins to decrease in (b) shows a similar trend to that in (a). Thus, even when different video is used, if the video (image) resolution, encoding method, and inference model used are predetermined, the change point at which the evaluation index mAP begins to decrease generally depends on the quantization parameter QP. Using this property, the quantization parameter derivation unit 34A of the control unit 34 derives the average quantization parameter QP per predetermined number of frames in the video, and the resolution determination unit 34B controls the processing resolution used for compression and encoding based on the derived average quantization parameter QP. As a result, in areas where the evaluation index mAP would decrease at the original resolution (the resolution before compression by the image encoding unit 32), the resolution is reduced to suppress the degree of decrease in the evaluation index mAP. On the other hand, in areas where the evaluation index mAP does not decrease at the original resolution, compression encoding is performed at the original resolution. As a result, the image encoding device 24 can maintain video quality.

[0035] 6A and 6B , the unit by which the control unit 34 switches the processing resolution will be described. In video coding standards such as H.264 / AVC and HEVC, the processing resolution can be changed in units of sequences. Therefore, the control unit 34 can set the processing resolution for each sequence.

[0036] 6A shows a case where the control unit 34 switches the resolution in units of a sequence, in which the quantization parameter derivation unit 34A of the control unit 34 derives an average quantization parameter QP for frames across the entire sequence. The resolution determination unit 34B determines the processing resolution for the next sequence in accordance with the derived average quantization parameter QP.

[0037] 6A, the quantization parameter derivation unit 34A derives an average quantization parameter QP for n frames F11 to F1n included in sequence 1. The resolution determination unit 34B compares the average quantization parameter QP with a threshold value, which will be described in detail later, and determines the processing resolution for frames F21 to F22 included in sequence 2. The example shown in Fig. 6A illustrates a case in which the resolution determination unit 34B determines, based on the average quantization parameter QP for frames F11 to F1n of sequence 1, that the processing resolution for sequence 2 is to be reduced compared to sequence 1.

[0038] On the other hand, in VVC, by applying an RPR (Reference Picture Resampling) encoding tool, the processing resolution can be changed on a frame-by-frame basis within a sequence. Therefore, when applying this encoding tool, the control unit 34 sets the processing resolution to be switched on a frame-by-frame basis, as shown in FIG. 6B . In this case, the control unit 34 derives an average quantization parameter QP for a previous frame temporally adjacent to the frame to be processed by the image encoding unit 32, or for multiple frames including the previous frame, and determines the processing resolution of the next frame based on the derived average quantization parameter QP. To further reduce the processing load on the control unit 34, it is also possible to derive an average quantization parameter QP on a larger GOP (Group of Pictures) basis and determine the processing resolution for the entire next GOP. In this way, when the processing resolution can be changed on a frame-by-frame basis within a sequence, it is possible to switch the processing resolution between frame-by-frame and units larger than frames. This switching may be set to be performed based on predetermined conditions, or may be performed in response to a user instruction or the like.

[0039] In the example shown in Figure 6B, the quantization parameter derivation unit 34A derives an average quantization parameter QP for each of n frames F11 to F1n included in sequence 1. The resolution determination unit 34B compares the average quantization parameter QP for each frame with a threshold value, which will be described in detail later, and determines the processing resolution for the next frame. The example shown in Figure 6B illustrates a case in which the resolution determination unit 34B determines to reduce the processing resolution of frame F13 based on the average quantization parameter QP of frame F12 in sequence 1, determines to reduce the processing resolution of frame F23 based on the average quantization parameter QP of frame F22 in sequence 2, and determines to reduce the processing resolution of frame F33 based on the average quantization parameter QP of frame F32 in sequence 3.

[0040] In this way, the control unit 34 can set the processing resolution in units according to the video encoding standard, and switch the processing resolution in the image encoding unit 32 .

[0041] The control unit 34 also sets the necessary operation modes and the like for each block within the image encoding device 24, accepts interrupts (not shown) from each block, and controls the operation timing of each block.

[0042] The external IF 42 is an interface for reading out the bit stream generated by the image encoding unit 32 and the inference result obtained by the inference processing unit 36 ​​from the memory 26 and sending them to the outside.

[0043] Next, a description will be given of the processing resolution control process executed by the control unit 34. Fig. 7 shows a flowchart illustrating an example of the flow of the processing resolution control process executed by the control unit 34. The control unit 34 starts the processing resolution control process shown in Fig. 7 by executing the resolution control processing program 27 when video converted by the preprocessing unit 30 into a format processable by the image encoding unit 32 is stored in the memory 26 as a processing target.

[0044] In step S100 of FIG. 7, the quantization parameter derivation unit 34A derives the average quantization parameter QP used in the compression encoding for each predetermined frame, as described above.

[0045] In the next step S102, the resolution determination unit 34B determines whether the currently set processing resolution reduction is OFF, i.e., whether the processing resolution has been reduced. If the processing resolution reduction is OFF, i.e., if the original processing resolution remains, the determination in step S102 is affirmative, and the process proceeds to step S104.

[0046] In step S104, the resolution determination unit 34B compares the average quantization parameter QP derived by the quantization parameter derivation unit 34A in step S100 with a predetermined threshold QPtho. The threshold QPtho used here corresponds to the quantization parameter QP at which the evaluation index mAP begins to decrease when the processing resolution reduction is OFF, i.e., at the original processing resolution, as described with reference to Fig. 5. In this embodiment, the user predetermines the threshold QPtho based on the desired accuracy, etc., through experiments, etc.

[0047] If the average quantization parameter QP is smaller than the threshold QPtho, the determination in step S104 is affirmative, and the process proceeds to step S106. In this case, it can be determined that the frame group from which the average quantization parameter QP was derived is an image that shows a tendency for the evaluation index mAP to be constant. Therefore, in step S106, the resolution determination unit 34B sets the processing resolution reduction to be maintained in the OFF state even in the next frame, and then proceeds to step S116. In other words, the process sets the processing resolution not to be reduced even in the next frame.

[0048] On the other hand, if the average quantization parameter QP is equal to or greater than the threshold QPtho, the determination in step S104 is negative, and the process proceeds to step S108. In this case, it can be determined that the frame group from which the average quantization parameter QP was derived is an image that shows a tendency for the evaluation index mAP to decrease.

[0049] Therefore, in step S108, the resolution determination unit 34B sets the processing resolution for the next frame to the reduction ON state, and then proceeds to step S116. That is, the processing resolution for the next frame is set to, for example, half the original processing resolution. By the control unit 34 setting the processing resolution in this way, the image processing unit 40 writes to memory an image converted to a state with a reduced processing resolution from the next frame onwards. As a result, the image encoding unit 32 performs compression encoding on the image (frame) with the reduced processing resolution.

[0050] The processing of step S108 will be further described with reference to FIG. 8A. FIG. 8A shows the change in the evaluation index mAP when the processing resolution setting is switched from the reduction-OFF state to the reduction-ON state. In FIG. 8A, the bit rate is the transmission bit rate, and the average quantization parameter QP derived by the quantization parameter derivation unit 34A is equal to or greater than the threshold QPtho. Therefore, if the original processing resolution is maintained, the evaluation index mAP may decrease. Therefore, the reduction is switched to the ON state from the next frame. As shown in FIG. 8A, by reducing the processing resolution, the decrease in the evaluation index mAP at the transmission bit rate is suppressed.

[0051] On the other hand, if the reduction of the processing resolution is currently set to ON, that is, if the processing resolution has been reduced, the determination in step S102 is affirmative, and the process proceeds to step S110.

[0052] In step S110, the resolution determination unit 34B compares the average quantization parameter QP derived by the quantization parameter derivation unit 34A in step S100 with a predetermined threshold QPths. As described with reference to FIG. 5, the threshold QPths used here is a quantization parameter QP that is expected to always fall below the threshold QPtho when the processing resolution reduction is ON, i.e., at a resolution reduced from the original processing resolution, assuming that the original resolution is obtained with the reduction turned OFF. In this embodiment, similar to the threshold QPtho described above, the user predetermines the threshold QPths through experiments or the like according to the desired accuracy, etc.

[0053] If the average quantization parameter QP is smaller than the threshold QPths, the determination in step S110 is affirmative, and the process proceeds to step S112.

[0054] In step S112, the resolution determination unit 34B sets the processing resolution for the next frame to the reduction OFF state, and then proceeds to step S116. That is, the processing resolution for the next frame is set to the original resolution. By the control unit 34 setting the processing resolution in this way, the image processing unit 40 writes images with the processing resolution set to the original resolution to memory from the next frame onwards. As a result, the image encoding unit 32 performs compression encoding on the images (frames) with the processing resolution set to the original resolution.

[0055] The process of step S112 will be further described with reference to FIG. 8B. FIG. 8B shows the change in the evaluation index mAP when the processing resolution setting is switched from the reduction-ON state to the reduction-OFF state. In FIG. 8B, the bit rate is the transmission bit rate, and the average quantization parameter QP derived by the quantization parameter derivation unit 34A is smaller than the threshold QPths. Therefore, the evaluation index mAP does not decrease even when the original resolution is restored. Therefore, the reduction is switched to the OFF state from the next frame.

[0056] On the other hand, if the average quantization parameter QP is equal to or greater than the threshold QPths, the determination in step S110 is negative, and the process proceeds to step S114. In this case, if downscaling is turned off and the resolution is restored to the original resolution for the frame group from which the average quantization parameter QP has been derived, the evaluation index mAP may decrease.

[0057] Therefore, in step S114, the resolution determination unit 34B maintains the processing resolution for the next frame in the reduction ON state, and then proceeds to step S116.

[0058] In step S116, the resolution determination unit 34B determines whether or not to end the resolution control process shown in Fig. 7. The determination in step S116 is negative, and the process returns to step S100, and the process of steps S100 to S114 is repeated until the process of steps S100 to S114 has been completed for all frames of the video input to the image encoding device 24. On the other hand, if the process of steps S100 to S114 has been completed for all frames of the video input to the image encoding device 24, the determination in step S116 is positive, and the resolution control process shown in Fig. 7 ends.

[0059] (Modification) Note that the technology of the present disclosure is not limited to the above-described embodiment. For example, in the above-described image encoding device 24, the type of switchable processing resolutions has been described as two types: the original resolution and a resolution half the original resolution. However, the type of switchable processing resolutions may be three or more. In this modification, a type in which the image encoding device 24 switches to one of three types of processing resolutions will be described. As a specific example, a type in which the image encoding device 24 can switch to three types of processing resolutions: a resolution reduction ratio of 1 / 1, a reduction ratio of 2 / 3, and a reduction ratio of 1 / 2 will be described.

[0060] In this modification, the configuration of the image encoding device 24 is the same as that of the image encoding device 24 in the above-described embodiment, and therefore a description thereof will be omitted. In this modification, as thresholds for determining the reduction ratio, resolution thresholds QPth11 and QPth12 when the reduction ratio is 1 / 1, resolution thresholds QPth21 and QPth22 when the reduction ratio is 2 / 3, and resolution thresholds QPth31 and QPth32 when the reduction ratio is 1 / 2 are used.

[0061] The threshold QPhth11 is a quantization parameter QP that is a change point at which the evaluation index mAP begins to decrease for video with a processing resolution reduction rate of 1 / 1. The threshold QPth12 is a quantization parameter QP that is a change point at which the evaluation index mAP begins to decrease more significantly than at the threshold QPth11 for video with a processing resolution reduction rate of 1 / 1. The threshold QPhth21 is a quantization parameter QP that is a change point at which the evaluation index mAP begins to decrease for video with a processing resolution reduction rate of 2 / 3. The threshold QPth22 is a quantization parameter QP that is a change point at which the evaluation index mAP begins to decrease more significantly than at the threshold QPth21 for video with a processing resolution reduction rate of 2 / 3. The threshold QPhth31 is a quantization parameter QP that is a change point at which the evaluation index mAP begins to decrease for video with a processing resolution reduction rate of 1 / 2. Furthermore, the threshold value QPth32 is a quantization parameter QP that is a change point at which the evaluation index mAP begins to drop significantly more significantly for video with a processing resolution reduced by 1 / 2 than at the threshold value QPth31. Similar to the threshold values ​​QPtho and QPths in the above embodiment, the threshold values ​​QPth11, QPth12, threshold values ​​QPth21, QPth22, and threshold values ​​QPhth31, QPth32 are determined in advance by the user through experiments or the like in accordance with the desired accuracy, etc.

[0062] FIG. 9 is a flowchart showing an example of the flow of the processing resolution control process executed by the control unit 34 of the image encoding device 24 of this modified example.

[0063] In step S200 of FIG. 9, the quantization parameter derivation unit 34A derives an average quantization parameter QP used in compression encoding for each predetermined frame, similar to step S100 of the processing resolution control process shown in FIG.

[0064] In the next step S202, the resolution determination unit 34B determines the reduction ratio of the currently set processing resolution. As described above, the resolution determination unit 34B of this modification determines whether the reduction ratio is 1 / 1, 2 / 3, or 1 / 2.

[0065] If the current reduction rate is 1 / 1, i.e., the original resolution, the process proceeds from step S202 to step S204. In step S204, the resolution determination unit 34B compares the average quantization parameter QP with a predetermined threshold QPth11. If the average quantization parameter QP is smaller than the threshold QPth11, the determination in step S204 is affirmative, and the process proceeds to step S206. In step S206, the resolution determination unit 34B sets the reduction rate of the processing resolution for the next frame to 1 / 1, and then the process proceeds to step S234.

[0066] On the other hand, if the average quantization parameter QP is equal to or greater than the threshold QPth11, the determination in step S204 is negative, and the process proceeds to step S208. In step S208, the resolution determination unit 34B compares the average quantization parameter QP with the threshold QPth12. If the average quantization parameter QP is smaller than the threshold QPth12, the determination in step S208 is positive, and the process proceeds to step S210. In step S210, the resolution determination unit 34B sets the reduction rate of the processing resolution for the next frame to ⅔, and then the process proceeds to step S234.

[0067] If the average quantization parameter QP is equal to or greater than the threshold QPth12, the determination in step S208 becomes negative, and the process proceeds to step S212. In step S212, the resolution determination unit 34B sets the reduction rate of the processing resolution for the next frame to 1 / 2, and then the process proceeds to step S234.

[0068] Furthermore, if the current reduction rate is 2 / 3, the process proceeds from step S202 to step S214. In step S214, the resolution determination unit 34B compares the average quantization parameter QP with a predetermined threshold QPth21. If the average quantization parameter QP is smaller than the threshold QPth21, the determination in step S214 is affirmative, and the process proceeds to step S216. In step S216, the resolution determination unit 34B sets the reduction rate of the processing resolution for the next frame to 1 / 1, and then the process proceeds to step S234.

[0069] On the other hand, if the average quantization parameter QP is equal to or greater than the threshold QPth21, the determination in step S214 is negative, and the process proceeds to step S218. In step S218, the resolution determination unit 34B compares the average quantization parameter QP with the threshold QPth22. If the average quantization parameter QP is smaller than the threshold QPth22, the determination in step S218 is positive, and the process proceeds to step S220. In step S220, the resolution determination unit 34B sets the reduction rate of the processing resolution for the next frame to ⅔, and then the process proceeds to step S234.

[0070] If the average quantization parameter QP is equal to or greater than the threshold QPth22, the determination in step S218 becomes negative, and the process proceeds to step S222. In step S222, the resolution determination unit 34B sets the reduction rate of the processing resolution for the next frame to 1 / 2, and then the process proceeds to step S234.

[0071] Furthermore, if the current reduction rate is 1 / 2, the process proceeds from step S202 to step S224. In step S224, the resolution determination unit 34B compares the average quantization parameter QP with a predetermined threshold QPth31. If the average quantization parameter QP is smaller than the threshold QPth31, the determination in step S224 is affirmative, and the process proceeds to step S226. In step S226, the resolution determination unit 34B sets the reduction rate of the processing resolution for the next frame to 1 / 1, and then the process proceeds to step S234.

[0072] On the other hand, if the average quantization parameter QP is equal to or greater than the threshold QPth31, the determination in step S224 is negative, and the process proceeds to step S228. In step S228, the resolution determination unit 34B compares the average quantization parameter QP with the threshold QPth32. If the average quantization parameter QP is smaller than the threshold QPth32, the determination in step S228 is positive, and the process proceeds to step S230. In step S230, the resolution determination unit 34B sets the reduction rate of the processing resolution for the next frame to ⅔, and then the process proceeds to step S234.

[0073] If the average quantization parameter QP is equal to or greater than the threshold QPth32, the determination in step S228 becomes negative, and the process proceeds to step S232. In step S232, the resolution determination unit 34B sets the reduction rate of the processing resolution for the next frame to 1 / 2, and then the process proceeds to step S234.

[0074] In step S234, the control unit 34 determines whether or not to end the resolution control process shown in Fig. 9, similar to step S116 of the processing resolution control process shown in Fig. 7. If the resolution control process is not to be ended, the determination in step S234 is negative, and the process returns to step S200, and the processes of steps S200 to S232 are repeated. On the other hand, if the resolution control process is to be ended, the determination in step S234 is positive, and the resolution control process shown in Fig. 9 is ended.

[0075] Thus, according to this modification, it is possible to control the processing resolution and inference accuracy more finely than in the above embodiment.

[0076] As described above, the image encoding device 24 includes the image encoding unit 32, the quantization parameter derivation unit 34A, and the resolution determination unit 34B.

[0077] The image encoding unit 32 performs compression encoding by compressing multiple input images set to a processing resolution using a lossy encoding method. The quantization parameter derivation unit 34A derives an average quantization parameter QP per frame for each predetermined number of frames. The resolution determination unit 34B determines the processing resolution by comparing the value of the average quantization parameter QP with at least one threshold value predetermined for each resolution.

[0078] As a result, the image encoding device 24 can suppress a decrease in inference accuracy on the cloud side during low bit rate transmission, while suppressing a decrease in video quality more than conventional techniques during high bit rate transmission. Therefore, the above-described image encoding device 24 can suppress a decrease in inference accuracy while also suppressing a decrease in video quality more than conventional techniques.

[0079] In addition, the various processes executed by the CPU after reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include dedicated electrical circuits, such as programmable logic devices (PLDs) whose circuit configuration can be changed after manufacture, such as field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs), which are processors having a circuit configuration designed specifically for executing specific processes. Furthermore, each of the learning process and the estimation process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0080] In addition, in each of the above embodiments, the resolution control processing program 27 is described as being pre-stored (installed) in the memory 26, but this is not limiting. The resolution control processing program 27 may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The resolution control processing program 27 may also be downloaded from an external device via a network.

[0081] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0082] The following additional clauses are disclosed in relation to the above-described embodiment.

[0083] (Supplementary Item 1) An image encoding device comprising: an image encoding unit that performs compression encoding by using a lossy encoding method to compress multiple input images included in a video having a processing resolution; a quantization parameter derivation unit that derives an average quantization parameter per frame for each predetermined number of frames; and a resolution determination unit that determines the processing resolution by comparing the value of the average quantization parameter with at least one threshold value that is predetermined for each resolution.

[0084] (Supplementary Item 2) The image encoding device described in Supplementary Item 1, wherein the threshold value is a value determined based on the inference accuracy when a quantization parameter corresponding to the threshold value is used to compress and decompress video at a processing resolution corresponding to the threshold value, and a predetermined inference process is performed on the decompressed video.

[0085] (Supplementary Item 3) The image encoding device according to Supplementary Item 1 or Supplementary Item 2, wherein the quantization parameter derivation unit derives an average quantization parameter per frame for each sequence in a video encoding standard during the compression by the image encoding unit, and the resolution determination unit determines the processing resolution for each sequence.

[0086] (Supplementary Item 4) The image encoding device according to Supplementary Item 1 or Supplementary Item 2, wherein the processing resolution can be changed on a frame-by-frame basis within a sequence in a video encoding standard during the compression by the image encoding unit, the quantization parameter derivation unit derives an average quantization parameter for a previous frame that is adjacent in time, or for a plurality of frames including the previous frame that is adjacent in time, and the resolution determination unit determines the processing resolution on a frame-by-frame basis.

[0087] (Supplementary Item 5) The image encoding device according to Supplementary Item 4, wherein the resolution determination unit is capable of switching the processing resolution between the frame unit and a unit larger than a frame.

[0088] (Supplementary Item 6) The image encoding device according to any one of Supplementary Items 1 to 5, further comprising an image processing unit that converts the resolution of the plurality of input images into the processing resolution.

[0089] (Supplementary Item 7) An image coding method comprising: performing compression coding to compress a plurality of input images included in a video having a processing resolution using a lossy coding method; deriving an average quantization parameter per frame for each predetermined number of frames; and determining the processing resolution by comparing the value of the average quantization parameter with at least one threshold value predetermined for each resolution.

[0090] (Supplementary Item 8) A program for causing a computer to function as the quantization parameter derivation unit and resolution determination unit of the image encoding device according to any one of Supplementary Items 1 to 6.

[0091] (Supplementary Item 9) An image encoding device comprising at least one processor, wherein the processor performs compression encoding to compress a plurality of input images included in a video having a processing resolution using a lossy encoding method, derives an average quantization parameter per frame for each predetermined number of frames, and determines the processing resolution by comparing the value of the average quantization parameter with at least one threshold value predetermined for each resolution.

[0092] (Supplementary Item 10) A non-transitory storage medium storing a program executable by a computer to execute a resolution control process for controlling the resolution when compressing and encoding an image, wherein the resolution control process is configured to: perform compression encoding by using a lossy encoding method to compress multiple input images included in a video set to a processing resolution; derive an average quantization parameter per frame for each predetermined number of frames; and determine the processing resolution by comparing the value of the average quantization parameter with at least one threshold value predetermined for each resolution.

[0093] 10 Event-driven inference system 20 Video transmission module 24 Image encoding device 27 Resolution control processing program 30 Preprocessing unit 32 Image encoding unit 34 Control unit, 34A Quantization parameter derivation unit, 34B Resolution determination unit 40 Image processing unit

Claims

1. An image encoding device comprising: an image encoding unit that performs compression encoding by compressing multiple input images included in a video having a processing resolution using a lossy encoding method; a quantization parameter derivation unit that derives an average quantization parameter per frame for each specified number of frames; and a resolution determination unit that determines the processing resolution by comparing the value of the average quantization parameter with at least one threshold value predetermined for each resolution.

2. The image encoding device of claim 1, wherein the threshold is a value determined based on the inference accuracy when compressing and decompressing an image at a processing resolution corresponding to the threshold using a quantization parameter equivalent to the threshold, and performing a predetermined inference process on the decompressed image.

3. The image encoding device according to claim 1, wherein the quantization parameter derivation unit derives an average quantization parameter per frame for each sequence in the video encoding standard used during the compression by the image encoding unit, and the resolution determination unit determines the processing resolution for each sequence.

4. The image encoding device of claim 1, wherein the processing resolution can be changed on a frame-by-frame basis within a sequence in a video encoding standard during the compression by the image encoding unit, the quantization parameter derivation unit derives an average quantization parameter for a previous frame that is adjacent in time, or for a plurality of frames including the previous frame that is adjacent in time, and the resolution determination unit determines the processing resolution on a frame-by-frame basis.

5. The image encoding device according to claim 4, wherein the resolution determination unit is capable of switching the processing resolution between the frame unit and a unit larger than a frame.

6. The image encoding device according to claim 1, further comprising an image processing unit that converts the resolution of the plurality of input images into the processing resolution.

7. An image coding method comprising: performing compression coding to compress a plurality of input images included in an image having a processing resolution using a lossy coding method; deriving an average quantization parameter per frame for each predetermined number of frames; and determining the processing resolution by comparing the value of the average quantization parameter with at least one threshold value predetermined for each resolution.

8. A program for causing a computer to function as the quantization parameter derivation unit and resolution determination unit of the image coding device according to any one of claims 1 to 6.

Images