Image encoding device, image encoding method, and program

Abstract

This image encoding device comprises: an image encoding unit that performs image encoding in which a plurality of input images are compressed by a lossy encoding method so as to reach a predetermined target code size; an inference processing unit that performs a predetermined inference process on the input image before the image encoding is carried out by the image encoding unit and on a local decoded image obtained by means of the image encoding performed by the image encoding unit, and outputs an inference result; and a control unit that changes the target code size on the basis of a plurality of inference results outputted by the inference processing unit.

Description

Image encoding device, image encoding method, and program

[0001] The disclosed technology relates to an image encoding device, an image encoding method, and a program.

[0002] There is a growing need for AI-based video analysis, and it is expected to be applied to various fields such as autonomous driving and surveillance. Generally, advanced AI-based video analysis requires a large amount of calculation, so attention is being paid to edge-cloud collaborative systems that achieve advanced processing by linking the cloud, which has fewer resource constraints and excellent scalability, with edge devices, which excel in real-time performance.

[0003] Non-Patent Document 1 proposes an event-driven inference system that detects events by performing AI inference processing with a relatively low computational load on the edge terminal, and the edge terminal transmits video to the cloud when the event is detected, where more advanced AI inference processing is performed. This makes it possible to further reduce unnecessary data transmission between the edge and the cloud and resource consumption in the cloud.

[0004] On the other hand, even in the event-driven inference system described above, if frequent events are detected on many edge devices, the transmission cost of transmitting video to the cloud can become an issue. Therefore, it is desirable for the edge device to compress and encode the video as much as possible before transmitting it, taking into account the decrease in inference accuracy in the cloud.

[0005] Non-Patent Document 2 proposes a technique for reducing video transmission bandwidth while maintaining inference accuracy on the cloud side as much as possible. The technique proposed in Non-Patent Document 2 measures mAP, a general evaluation index for object detection accuracy, for decoded video after video transmission, and an encoder controls the code rate according to the mAP value, thereby reducing the video transmission bandwidth as much as possible while maintaining the video evaluation index mAP at a desired value. Specifically, Non-Patent Document 2 measures mAP (Mean Average Precision), a general evaluation index for object detection, and controls CRF (Constant Rate Factor), an H.265 video quality index, based on the measurement result so that mAP is always constant. The smaller the value of the video quality index CRF, the higher the quality and the higher the bit rate. In Non-Patent Document 2, for example, if the current mAPcur does not satisfy the requested value mAPreq (mAPcur<mAPreq), it is determined that the current CRFcur does not satisfy the requested mAPreq, and CRFnext in the next frame is controlled to be slightly smaller (CRFnext=CRFcur-β), thereby improving video quality. In Non-Patent Document 2, by dynamically controlling video quality (CRF) for each frame in this way, the video transmission bandwidth is reduced as much as possible while maintaining the desired accuracy (mAPreq).

[0006] Eda, et al., “Highly Efficient Event-Driven Inference for AI Services in the IOWN Era,” NTT Technical Journal, 2020; Ito, et al., “Adaptive Video Quality Control Methods for Object Detection,” IEICE General Conference, 2022

[0007] In order to control the video quality index CRF while measuring the video assessment index mAP in order to maintain the desired accuracy as in the technology described in Non-Patent Document 2, it is necessary to perform inference processing of the original image, compression and decompression processing of the image, and inference processing of the image after compression and decompression on the edge terminal. This increases the amount of calculation on the edge terminal, which may result in a decrease in processing speed and an increase in hardware size.

[0008] The technology disclosed herein has been made in consideration of the above points, and provides an image coding device, an image coding method, and a program that can reduce transmission bandwidth while maintaining desired accuracy and suppressing an increase in the amount of calculation.

[0009] The image encoding device of the present disclosure includes an image encoding unit that performs image encoding by compressing multiple input images using a lossy encoding method to a predetermined target code size, an inference processing unit that performs a predetermined inference process on the input images before the image encoding by the image encoding unit and on locally decoded images obtained during the image encoding process by the image encoding unit, and outputs the inference results, and a control unit that changes the target code size based on the multiple inference results output by the inference processing unit.

[0010] In addition, the image encoding method disclosed herein performs image encoding to compress multiple input images using a lossy encoding method to a predetermined target code size, performs a predetermined inference process on the input images before the image encoding and on locally decoded images obtained during the image encoding process, outputs the inference results, and changes the target code size based on the multiple inference results.

[0011] Furthermore, the program of the present disclosure is for causing a computer to function as a control unit of the image encoding device of the present disclosure.

[0012] According to the technology of the present disclosure, it is possible to reduce the transmission band while maintaining the desired accuracy and suppressing an increase in the amount of calculation.

[0013] FIG. 1 is a configuration diagram showing an example of the configuration of an event-driven inference system of an embodiment. FIG. 2 is a configuration diagram showing an example of the configuration of a video transmission module of an embodiment. FIG. 3 is a block diagram showing an example of the configuration of an image encoding device of an embodiment. FIG. 4 is a diagram for explaining the flow of data in the image encoding device of an embodiment. FIG. 5 is a flowchart showing an example of the flow of feedback processing executed in the control unit of an embodiment. FIG. 6 is a diagram showing an example of a reference structure of each frame that is common in video encoding methods. FIG. 7 is a diagram for explaining an example of a method for deriving a video evaluation index from the inference result of an inference processing unit. FIG. 8 is a diagram for explaining an example of a method for deriving a video quality index from a video evaluation index.

[0014] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals.

[0015] First, an example of the configuration of an event-driven inference system according to this embodiment will be described. Fig. 1 shows an example of an event-driven inference system 10 according to this embodiment.

[0016] The event-driven inference system 10 shown in FIG. 1 includes a cloud 12 and an edge terminal group 15. The edge terminal group 15 includes a plurality of edge terminals 14, and in the present embodiment, as an example shown in FIG. 1 , includes a surveillance camera 14A, a drone 14B, an automobile 14C, and a mobile terminal 14D. Note that the number and types of edge terminals 14 included in the edge terminal group 15 are not limited to this embodiment. Note that in the present embodiment, the surveillance camera 14A, the drone 14B, the automobile 14C, and the mobile terminal 14D included in the edge terminal group 15 are collectively referred to as "edge terminals 14" when not distinguishing between them individually.

[0017] Each edge terminal 14 is wirelessly connected to the cloud 12 via a network (not shown). The edge terminal 14 detects a predetermined event by performing a predetermined inference process, and transmits video to the cloud 12 at the timing of detecting the event. Note that in this embodiment, "video" refers to a moving image including multiple images (frames).

[0018] The cloud 12 is a so-called cloud computing system, and is constructed in a data center or the like. The cloud 12 performs more advanced inference processing on the video obtained from the edge terminal 14. The inference results obtained by the inference processing in the cloud 12 can be used in other service applications 4. The cloud 12 can also control the operation of the edge terminal 14 by sending instructions to the edge terminal 14 according to the obtained inference results.

[0019] The edge terminal 14 of this embodiment performs predetermined inference processing on video as described above, and includes a video transmission module 20, an example of which is shown in Fig. 2. The video transmission module 20 shown in Fig. 2 includes a camera 22, an image encoding device 24, a memory 26, and a wireless module 28.

[0020] The camera 22 is an example of a device for acquiring video in the edge terminal 14. The camera 22 outputs the video obtained by capturing the video to the image encoding device 24. Note that, in this embodiment, a form in which the image encoding device 24 processes the video obtained by the camera 22 will be described, but the image to be processed by the image encoding device 24 is not limited to the video obtained by the camera 22. For example, the image may be a moving image such as an animation composed of multiple images drawn by a user in the edge terminal 14. In this embodiment, the video input from the camera 22 to the image encoding device 24 is an example of the multiple input images of the present disclosure.

[0021] The image encoding device 24 is a device, such as an accelerator, for performing image encoding and predetermined inference processing on the video input from the camera 22. The image encoding device 24 outputs the inference results obtained by the inference processing and the bit stream obtained by the image encoding to the memory 26. The wireless module 28 reads the bit stream stored in the memory 26 and transmits it to the cloud 12 when an event is detected, based on instructions from a control unit 34 (see FIG. 3 ) provided in the image encoding device 24.

[0022] The memory 26 stores the bitstream and inference results obtained by the image encoding device 24. The memory 26 also stores an image encoding processing program 27 executed by the image encoding device 24. The image encoding processing program 27 may be a single program or a group of programs configured by multiple programs or modules. The memory 26 is configured by an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The image encoding processing program 27 of this embodiment is an example of a program of the present disclosure.

[0023] Next, the image encoding device 24 of this embodiment will be described in detail. Fig. 3 shows a block diagram illustrating an example of the configuration of the image encoding device 24. As shown in Fig. 3, the image encoding device 24 of this embodiment includes a preprocessing unit 30, an image encoding unit 32, a control unit 34, an inference processing unit 36, a memory IF (Interface) 38, an image processing unit 40, and an external IF 42. Each component is connected to each other so as to be able to communicate with each other via a bus 49 such as a system bus or a control bus.

[0024] The pre-processing unit 30 converts the video input from the camera 22 into a format (here, YUV420) that can be processed by the image encoding unit 32, and writes the converted video to the memory 26 via the memory IF 38. As an example, the pre-processing unit 30 of this embodiment converts the video input from the camera 22 into the YUV420 format. The pre-processing unit 30 also converts the video input from the camera 22 into a format that can be processed by the inference processing unit 36, and writes the converted video to the memory 26. As an example, the pre-processing unit 30 of this embodiment converts the video input from the camera 22 into the RGB format. In this embodiment, an image before being encoded by the image encoding unit 32 is called an "original image."

[0025] The image encoding unit 32 performs image encoding by compressing the input video using a lossy encoding method to a predetermined target code size. The image encoding unit 32 in this embodiment is a so-called encoder. The image encoding unit 32 reads the video written to the memory 26, performs compression encoding in accordance with a video encoding standard such as H.264 / AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding), and writes the resulting bitstream to the memory 26. The image encoding unit 32 also references a video quality index determined by the control unit 34 and controls the code size in accordance with the video quality index. In currently widely used video coding methods, a high compression rate can be achieved by calculating the difference between an input image and a predicted image generated within the image coding unit 32 and entropy coding the difference. In this case, the image coding unit 32 typically generates a locally decoded image that is identical to the decoded image, and then generates a highly accurate predicted image based on this locally decoded image and taking motion vectors into account. In this embodiment, too, the image coding unit 32 generates a locally decoded image, and writes the generated locally decoded image to the memory 26.

[0026] The image processing unit 40 reads the locally decoded image from the memory 26, converts it into a format processable by the inference processing unit 36, and writes the converted image to the memory 26. As a result, both the original image input from the camera and the locally decoded image generated by the image encoding unit 32 are converted into a format processable by the inference processing unit 36. This allows the inference processing unit 36 ​​to perform inference processing on both images. In this way, the image encoding device 24 of this embodiment utilizes the characteristic that the decoded image and the locally decoded image generated by the image encoding unit 32 are exactly the same, so that an inference result for the decoded image can be obtained using only the image encoding unit 32, i.e., the encoder, without having to re-decode the bitstream generated by the image encoding unit 32 to reconstruct the image.

[0027] In this embodiment, the image processing unit 40 is provided separately from the image encoding unit 32 and the inference processing unit 36, but the image encoding unit 32 and the inference processing unit 36 ​​may have functions related to the format conversion performed by the image processing unit 40. For example, the inference processing unit 36 ​​may read the locally decoded image written to the memory 26 by the image encoding unit 32 while performing format conversion, thereby further reducing the amount of access to the memory 26. Furthermore, if the input image of the inference processing model executed by the inference processing unit 36 ​​is learned assuming YUV420, or if format conversion of the locally decoded image is not required, the image processing unit 40 may not be provided.

[0028] The inference processing unit 36 ​​performs predetermined inference processing on the video before image encoding by the image encoding unit 32 and the locally decoded image obtained in the process of image encoding by the image encoding unit 32, and outputs the inference result. The inference processing unit 36 ​​of this embodiment is an AI (Artificial Intelligence) inference core that has been trained in advance by a neural network, and writes the inference result for each of the original image and the locally decoded image for a predetermined frame to the memory 26.

[0029] The control unit 34 changes the target code amount in the image encoding unit 32 based on the multiple inference results output by the inference processing unit 36. In this embodiment, a CPU (Central Processing Unit) is used as the control unit 34. The control unit 34 in this embodiment derives a video evaluation index mAP based on the multiple inference results, further derives a video quality index CRF from the video evaluation index mAP, and notifies the image encoding unit 32 of the derived video quality index CRF, thereby causing the image encoding unit 32 to change the target code amount.

[0030] The control unit 34 of this embodiment determines a video quality index to be used for controlling the code amount of the image encoding unit 32 based on the inference result obtained from the inference processing unit 36, and notifies the image encoding unit 32. As an example, in this embodiment, a CRF (Constant Rate Factor) is used as the video quality index as in the technology described in Non-Patent Document 2, but other video quality indexes such as a QP (Quantize Parameter) or a transmission bit rate (bps) may also be used.

[0031] The control unit 34 also sets the necessary operation modes and the like for each block within the image encoding device 24, accepts interrupts (not shown) from each block, and controls the operation timing of each block.

[0032] The external IF 42 is an interface for reading out the bit stream generated by the image encoding unit 32 and the inference result obtained by the inference processing unit 36 ​​from the memory 26 and sending them to the outside.

[0033] 4, the flow of data in the preprocessing unit 30, image encoding unit 32, control unit 34, inference processing unit 36, and image processing unit 40 will be described. Note that the memory 26 is not shown in FIG.

[0034] As described above, the pre-processing unit 30 converts the format of the video input from the camera 22 into YUV420, which can be processed by the inference processing unit 36. The original image, whose format has been converted into YUV420, is input to the inference processing unit 36. The pre-processing unit 30 also converts the format of the video input from the camera 22 into RGB, which can be processed by the image encoding unit 32. The original image, whose format has been converted into RGB, is input to the image encoding unit 32.

[0035] The image encoding unit 32 generates a locally decoded image in the process of compression-encoding the original image so as to obtain a video quality index (CRF). The image processing unit 40 converts the format of the locally decoded image generated by the image encoding unit 32 into YUV420, which can be processed by the inference processing unit 36. The locally decoded image whose format has been converted into YUV420 is input to the inference processing unit 36.

[0036] The inference processing unit 36 ​​switches between inputting an original image and a locally decoded image at a predetermined timing for processing. This switching, i.e., the setting of the image input to the inference processing unit 36, is performed by the control unit 34. Specifically, the control unit 34 switches the address at which the inference processing unit 36 ​​reads the input image from the memory 26, thereby setting the image input to the inference processing unit 36 ​​as either an original image or a locally decoded image. For example, the control unit 34 can instruct the inference processing unit 36 ​​to process the locally decoded image approximately once every 30 frames. By switching the image to be processed by the inference processing unit 36 ​​on a frame-by-frame basis in this manner, the required performance, computational complexity, hardware size, etc., of the inference processing unit 36 ​​can be reduced. After the inference processing unit 36 ​​obtains inference results for both the original image and the locally decoded image for a predetermined frame, the control unit 34 evaluates the accuracy of the inference result for the locally decoded image when the inference result for the original image is considered as true ground truth data (Ground Truth) and derives a video evaluation index. The control unit 34 also derives a video quality index CRF from the obtained video evaluation index and notifies the image encoding unit 32 of the derived video quality index CRF. In this embodiment, mAP (Mean Average Precision) is used as the video evaluation index, but the video evaluation index is not limited to mAP. For example, IoU (Intersecting Union), AP (Average Precision), AR (Average Recall), etc. may also be used as the video evaluation index.

[0037] In this way, the control unit 34 measures the inference accuracy and feeds back the measurement results to the image encoding unit 32. These controls make it possible to transmit video at the minimum necessary bit rate while suppressing deterioration in the inference accuracy of the inference performed on the cloud 12 side after video transmission of the bit stream output by the image encoding unit 32.

[0038] Next, a description will be given of the feedback process of the video quality index CRF for the image encoding process in the image encoding unit 32, which is executed by the control unit 34. Fig. 5 shows a flowchart illustrating an example of the flow of the feedback process executed by the control unit 34. The control unit 34 starts the feedback process shown in Fig. 5 by executing the image encoding process program 27 when either the original image or the locally decoded image is stored in the memory 26 as a target for processing by the inference processing unit 36 ​​and is ready to be input to the inference processing unit 36.

[0039] 5, the control unit 34 determines whether or not preparation of the image to be processed in the inference processing unit 36 ​​is complete. The determination in step S100 remains negative until at least one of the original image converted into a format (YUV420) that can be processed by the inference processing unit 36 ​​and the locally decoded image is stored in the memory 26. On the other hand, if preparation of the image to be processed is complete, the determination in step S100 remains positive, and the process proceeds to step S102.

[0040] In step S102, the control unit 34 determines whether or not the locally decoded image can be used. An example of a method by which the control unit 34 determines whether or not the locally decoded image can be used will be described with reference to FIG. 6 . FIG. 6 shows a reference structure (random access structure) of each frame (picture) that is common in video encoding methods such as HEVC. FIG. 6 shows a GOP (Group of Pictures) consisting of nine frames, frames 50_0 to 50_8, with arrows between each frame indicating the reference relationship between the frames. In the example shown in FIG. 6 , frames 50_1, 50_3, 50_5, and 50_7 are not referenced by any other frames, and therefore, typically, locally decoded images are not generated during the image encoding process by the image encoding unit 32. Therefore, the control unit 34 of this embodiment determines whether or not the locally decoded image can be used based on a predetermined reference structure of each frame, etc.

[0041] If the locally decoded image is not available, the determination in step S102 is negative, and the process proceeds to step S106. On the other hand, if the locally decoded image is available, the determination in step S102 is positive, and the process proceeds to step S104.

[0042] In step S104, the control unit 34 determines whether or not a predetermined timing has occurred. For example, if the predetermined timing, such as once every 30 frames, has not yet occurred, the determination in step S104 is negative, and the process proceeds to step S106. In step S106, the control unit 34 sets the original image in YUV420 format stored in the memory 26 as the input to the inference processing unit 36, and then the process proceeds to step S110.

[0043] On the other hand, if the predetermined timing has been reached, the determination in step S104 becomes positive, and the process proceeds to step S108. In step S108, the control unit 34 sets the locally decoded image in YUV420 format stored in the memory 26 as the input to the inference processing unit 36, and then the process proceeds to step S110.

[0044] In step S110, the control unit 34 starts the inference processing in the inference processing unit 36. As a result, the inference processing unit 36 ​​performs inference processing of the original image or the locally decoded image by the AI ​​inference core.

[0045] In the next step S112, the control unit 34 determines whether the inference processing in the inference processing unit 36 ​​has finished. Until the inference processing has finished, the determination in step S112 remains negative. On the other hand, when the inference processing has finished, the determination in step S112 remains positive, and the process proceeds to step S114.

[0046] In step S114, the control unit 34 determines whether or not it is possible to derive the video evaluation index mAP. If it is not possible to derive the video evaluation index mAP, the determination in step S114 is negative, and the process returns to step S100, and the processes in steps S100 to S112 are repeated. On the other hand, if it is possible to derive the video evaluation index mAP, the determination in step S114 is positive, and the process proceeds to step S116.

[0047] In step S116, the control unit 34 derives a video evaluation index mAP from the inference result of the inference processing unit 36. An example of a method by which the control unit 34 derives a video evaluation index mAP from the inference result of the inference processing unit 36 ​​will be described with reference to FIG. 7 . FIG. 7 shows the data structure of the inference result written to the memory 26 by the inference processing unit 36. The inference processing unit 36 ​​writes the inference result to the memory 26, linking it with a frame ID for identifying the frame to be processed, so that the control unit 34 can uniquely identify the inference result. In the example shown in FIG. 7 , the control unit 34 instructs the inference processing unit 36 ​​to input a locally decoded image at the time of the frame with frame ID 2, and an inference result C for the original image and an inference result D for the locally decoded image are stored in the memory 26. The control unit 34 compares the inference result C for the original image with the inference result D for the local image for the frame with frame ID 2. Specifically, the control unit 34 evaluates the accuracy of the inference result D when the inference result C for the original image is set as correct data for the frame with frame ID 2. In this embodiment, the AI ​​inference core used by the inference processing unit 36 ​​for inference is assumed to be capable of outputting only one frame's worth of inference results per unit time. Therefore, to obtain an inference result for the original image and an inference result for the local image for a given frame, a processing time equivalent to two frames is required, and the processing time for the next frame must be thinned out. In the example shown in FIG. 7 , to obtain an inference result C for the original image and an inference result D for the local image for the frame with frame ID 2, a processing time equivalent to two frames is required. Since processing of the frame with frame ID 3 is omitted, both the original image and the inference result are "-". Note that the method by which the control unit 34 derives the video evaluation index mAP from the inference result of the inference processing unit 36 ​​is not limited to this method.

[0048] In the next step S118, the control unit 34 derives a video quality index CRF from the video evaluation index mAP and notifies the image encoding unit 32. An example of a method by which the control unit 34 derives the video quality index CRF from the video evaluation index mAP will be described with reference to FIG. 8 . FIG. 8 shows a table representing the correspondence between the video evaluation index mAP derived by the control unit 34 and the video quality index CRF notified by the control unit 34 to the image encoding unit 32. The table representing the correspondence is stored in advance in the memory 26 or the like and can be referenced by the control unit 34. By referencing the table representing the correspondence, the control unit 34 can derive the video quality index CRF based on the value of the video evaluation index mAP. Note that the method by which the control unit 34 derives the video quality index CRF from the video evaluation index mAP is not limited to the method of referencing a table representing the correspondence between the video evaluation index mAP and the video quality index CRF. For example, the method described in Non-Patent Document 2 may be used. Alternatively, for example, a method may be used in which the relationship between the video evaluation index mAP and the video quality index CRF is approximated in advance by a function and the CRF is derived using the function.

[0049] Upon receiving the notification of the video quality index CRF through the processing of this step, the video encoding unit 32 controls the target code amount when encoding the video in accordance with the video quality index CRF.

[0050] In the next step S120, the control unit 34 determines whether or not to end this feedback process. The determination in step S120 is negative, and the process returns to step S100, and the process in steps S100 to S118 is repeated until the processes in steps S100 to S118 have been completed for all frames of the video input to the image encoding device 24. On the other hand, if the processes in steps S100 to S118 have been completed for all frames of the video input to the image encoding device 24, the determination in step S120 is positive, and the feedback process shown in FIG. 5 ends.

[0051] As described above, the image encoding device 24 includes an image encoding unit 32, a control unit 34, and an inference processing unit 36. The image encoding unit 32 performs image encoding, compressing video using a lossy encoding method to a predetermined target code size. The inference processing unit 36 ​​performs predetermined inference processing on original images before image encoding by the image encoding unit 32 and locally decoded images obtained during image encoding by the image encoding unit 32, and outputs the inference results. The control unit 34 derives a video evaluation index mAP based on the multiple inference results output by the inference processing unit 36, and notifies the image encoding unit 32 of a video quality index CRF derived from the video evaluation index mAP, thereby changing the target code size.

[0052] As a result, in the image encoding device 24, the control unit 34 can control the target code amount of the image encoding unit 32 from the original image and the local image without requiring a decoder. Therefore, according to the above-described image encoding device 24, it is possible to reduce the transmission bandwidth when transmitting video from the edge terminal 14 to the cloud 12 while maintaining the desired accuracy and suppressing an increase in the amount of calculation.

[0053] In addition, the various processes executed by the CPU after reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include dedicated electrical circuits, such as programmable logic devices (PLDs) whose circuit configuration can be changed after manufacture, such as field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs), which are processors having a circuit configuration designed specifically for executing specific processes. Furthermore, each of the learning process and the estimation process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0054] In addition, in each of the above embodiments, the image encoding processing program 27 is described as being pre-stored (installed) in the memory 26, but this is not limiting. The image encoding processing program 27 may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. Furthermore, the image encoding processing program 27 may be downloaded from an external device via a network.

[0055] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[0056] The following additional clauses are disclosed in relation to the above-described embodiment.

[0057] (Additional Item 1) An image encoding device comprising: an image encoding unit that performs image encoding to compress a plurality of input images using a lossy encoding method so that the compressed images reach a predetermined target code size; an inference processing unit that performs a predetermined inference process on the input images before the image encoding by the image encoding unit and on a locally decoded image obtained during the image encoding process by the image encoding unit, and outputs an inference result; and a control unit that changes the target code size based on the plurality of inference results output by the inference processing unit.

[0058] (Supplementary Item 2) The image encoding device according to Supplementary Item 1, wherein the control unit changes the target code amount in accordance with an inference result for the input image before the image encoding is performed and an inference result for the locally decoded image.

[0059] (Supplementary Item 3) The image encoding device according to Supplementary Item 1 or Supplementary Item 2, wherein the control unit switches the processing target of the inference processing unit between the input image and the locally decoded image at a predetermined timing.

[0060] (Supplementary Item 4) The image encoding device according to any one of Supplementary Items 1 to 3, further comprising: a pre-processing unit that converts the input image into a form that can be processed by the inference processing unit; and an image processing unit that converts the locally decoded image into a form that can be processed by the inference processing unit, wherein the inference processing unit performs the inference processing on the input image after conversion by the pre-processing unit and the locally decoded image after conversion by the image processing unit.

[0061] (Supplementary Item 5) The image encoding device described in any one of Supplementary Items 1 to 4, wherein the control unit derives a video evaluation index based on multiple inference results output by the inference processing unit, further derives a video quality index from the video evaluation index, and notifies the image encoding unit of the video quality index, thereby causing the image encoding unit to change the target code amount.

[0062] (Supplementary Item 6) An image coding method comprising: performing image coding to compress a plurality of input images using a lossy coding method so that the input images reach a predetermined target code size; performing a predetermined inference process on the input images before the image coding and on locally decoded images obtained during the image coding process; outputting an inference result; and changing the target code size based on a plurality of the inference results.

[0063] (Supplementary Item 7) A program for causing a computer to function as a control unit of the image encoding device according to any one of Supplementary Items 1 to 5.

[0064] (Supplementary Item 8) An image encoding device comprising at least one processor, the processor performs image encoding to compress a plurality of input images using a lossy encoding method to a predetermined target code size, performs a predetermined inference process on the input images before the image encoding and on locally decoded images obtained during the image encoding process, outputs an inference result, and changes the target code size based on a plurality of the inference results.

[0065] (Supplementary Item 9) A non-transitory storage medium storing a program executable by a computer to execute feedback processing for image encoding, wherein the feedback processing is configured to: perform image encoding to compress a plurality of input images using a lossy encoding method so that the input images reach a predetermined target code size; perform a predetermined inference process on the input images before the image encoding and on locally decoded images obtained during the image encoding process to obtain inference results; and change the target code size based on a plurality of the inference results.

[0066] REFERENCE SIGNS LIST 10 Event-driven inference system 20 Video transmission module 24 Image encoding device 27 Image encoding processing program 30 Preprocessing unit 32 Image encoding unit 34 Control unit 36 ​​Inference processing unit 40 Image processing unit

Claims

1. An image encoding device comprising: an image encoding unit that performs image encoding by compressing multiple input images using a lossy encoding method so that the encoded images reach a predetermined target code size; an inference processing unit that performs predetermined inference processing on the input images before the image encoding by the image encoding unit and on locally decoded images obtained during the image encoding process by the image encoding unit, and outputs the inference results; and a control unit that changes the target code size based on the multiple inference results output by the inference processing unit.

2. The image encoding device according to claim 1, wherein the control unit changes the target code amount in accordance with an inference result for the input image before the image encoding is performed and an inference result for the locally decoded image.

3. The image encoding device according to claim 1, wherein the control unit switches the processing target of the inference processing unit between the input image and the locally decoded image at a predetermined timing.

4. The image encoding device of claim 1, further comprising: a pre-processing unit that converts the input image into a form that can be processed by the inference processing unit; and an image processing unit that converts the locally decoded image into a form that can be processed by the inference processing unit, wherein the inference processing unit performs the inference processing on the input image after conversion by the pre-processing unit and the locally decoded image after conversion by the image processing unit.

5. The image encoding device of claim 1, wherein the control unit derives a video evaluation index based on multiple inference results output by the inference processing unit, further derives a video quality index from the video evaluation index, and notifies the image encoding unit of the video quality index, thereby causing the image encoding unit to change the target code amount.

6. An image coding method comprising: performing image coding to compress a plurality of input images using a lossy coding method so that the compressed images reach a predetermined target code size; performing a predetermined inference process on the input images before the image coding and on locally decoded images obtained during the image coding process; outputting an inference result; and changing the target code size based on the plurality of inference results.

7. A program for causing a computer to function as a control unit of the image encoding device according to any one of claims 1 to 5.

Images