Image encoding device and image encoding method

Abstract

This image encoding device includes: a detection unit for detecting an object in each frame constituting a moving image; an evaluation unit for evaluating whether or not each region in each frame is a region of interest using detection results from the detection unit; a parameter determination unit for determining an encoding parameter for controlling image quality for each region in the frame; a correction unit for correcting, for a region determined to be the region of interest among each region in the frame, the encoding parameter on the basis of the size of an object corresponding to the region; and an encoding unit for encoding each region in the frame using the encoding parameter.

Description

Image encoding device and image encoding method

[0001] The technology of the present disclosure relates to an image encoding device and an image encoding method.

[0002] There is a growing need for video analysis using AI (Artificial Intelligence), and it is expected to be applied to a variety of fields, including autonomous driving and surveillance. Generally, advanced video analysis using AI requires a large amount of calculation. For this reason, attention is being paid to edge-cloud collaborative systems, which enable advanced processing by linking clouds, which have fewer resource constraints and excellent scalability, with edge devices, which excel in real-time performance.

[0003] An event-driven inference system has been proposed in which events are detected by performing AI inference processing with a relatively low computational load on the edge device, and the edge device transmits the video to the cloud when the event is detected, where more advanced AI inference processing is performed. This can further reduce unnecessary data transmission between the edge and cloud and resource consumption in the cloud.

[0004] On the other hand, even in the above-mentioned event-driven inference system, if frequent events are detected on a large number of edge terminals, the transmission cost of transmitting video to the cloud can become an issue. Therefore, it is desirable for the edge terminal to compress and transmit video as much as possible, taking into account the decline in inference accuracy on the cloud. The technology described in Non-Patent Document 1 has been proposed as a method for reducing video transmission bandwidth while maintaining as much inference accuracy on the cloud side as possible. In the technology described in Non-Patent Document 1, regions of interest (ROIs) important for AI processing are detected using an object detector such as YOLO (You Look Only Once), and the QP (quantization parameter) is uniformly reduced for the ROI compared to non-ROIs when encoding video. In other words, by lowering the compression rate only for regions important for AI processing and increasing the compression rate for other regions, the bit rate is reduced while maintaining the AI ​​inference accuracy for the encoded video.

[0005] Kristian Fischer, etc. “Saliency-Driven Versatile Video Coding for Neural Object Detection”, 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021

[0006] Experiments by the inventors have confirmed that as the transmission bit rate decreases, the degree of decrease in inference accuracy varies depending on the size of the object, such as the degree of decrease in AI inference accuracy for relatively small objects in the video being greater than for large objects. Therefore, if QP control is performed uniformly regardless of object size as described in Non-Patent Document 1, sufficient inference accuracy may not be obtained depending on the size of the object.

[0007] The disclosed technology has been developed in consideration of the above points, and aims to provide an image encoding device and an image encoding method that can compress data while suppressing a decrease in inference accuracy depending on the size of the object.

[0008] A first aspect of the present disclosure is an image encoding device including: a detection unit that detects an object in each frame that constitutes a moving image; a determination unit that determines whether each region in each frame is an area of ​​interest using the detection result of the detection unit; a parameter determination unit that determines encoding parameters for controlling the image quality of each region in the frame; a modification unit that modifies the encoding parameters for each region in the frame that is determined to be an area of ​​interest based on the size of the object corresponding to the region; and an encoding unit that encodes each region in the frame using the encoding parameters.

[0009] A second aspect of the present disclosure is an image encoding method executed by a computer to detect an object in each frame constituting a moving image, determine whether each region in each frame is an area of ​​interest using the detection results, determine encoding parameters for controlling image quality for each region in the frame, modify the encoding parameters for each region in the frame that is determined to be an area of ​​interest based on the size of the object corresponding to the region, and encode each region in the frame using the encoding parameters.

[0010] According to the disclosed technology, it is possible to compress data while suppressing the decrease in inference accuracy according to the size of the object.

[0011] 1 is a block diagram showing the configuration of an event-driven inference system of this embodiment. FIG. 2 is a block diagram showing the configuration of a video transmission module included in an edge terminal of this embodiment. FIG. 3 is a schematic block diagram of an example of a computer functioning as an image encoding device of this embodiment. FIG. 4 is a block diagram showing the functional configuration of the image encoding device of this embodiment. FIG. 5 is a block diagram showing the functional configuration of a control unit of the image encoding device of this embodiment. FIG. 6 is a graph showing the relationship between an encoding parameter QP and an evaluation index AP for each object size. FIG. 7 is a graph showing the relationship between an encoding parameter QP and a degree of decline in the evaluation index AP for each object size. FIG. 8 is a diagram showing an example of setting a correction amount DeltaQP of the encoding parameter QP. FIG. 9 is a diagram showing an example of setting a correction amount DeltaQP of the encoding parameter QP. FIG. 10 is a diagram for explaining a method of determining an area of ​​interest. FIG. 11 is a flowchart showing the flow of image encoding processing of this embodiment. FIG. 12 is a flowchart showing the flow of processing for determining an area of ​​interest of this embodiment. FIG. 13 is a diagram showing an example of an area of ​​interest determination result. FIG. 14 is a diagram showing an example of an area of ​​interest determination result. FIG. 15 is a diagram showing an example of a processing for correcting encoding parameters of this embodiment. FIG. 16 is a diagram showing an example of a determination table.

[0012] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0013] <Configuration of an Event-Driven Inference System According to the Present Embodiment> The event-driven inference system 100 shown in Figure 1 includes a cloud 22 and a group of edge terminals. The group of edge terminals includes a plurality of edge terminals 20, and in the present embodiment, as an example, as shown in Figure 1, includes a surveillance camera 20A, a drone 20B, an automobile 20C, and a mobile terminal 20D. Note that the number and types of edge terminals 20 included in the group of edge terminals are not limited to this embodiment. Note that in the present embodiment, the surveillance camera 20A, the drone 20B, the automobile 20C, and the mobile terminal 20D included in the group of edge terminals are collectively referred to as "edge terminals 20" when not distinguishing between them individually.

[0014] Each edge terminal 20 is wirelessly connected to the cloud 22 via a network (not shown). The edge terminal 20 detects a predetermined event by performing a predetermined inference process, and transmits video to the cloud 22 at the timing of detecting the event. In this embodiment, "video" refers to a moving image including multiple images (frames).

[0015] The cloud 22 is a so-called cloud computing system, and is constructed in a data center or the like. The cloud 22 performs more advanced inference processing on the video obtained from the edge terminal 20. The inference results obtained by the inference processing in the cloud 22 can be used in other services and applications. The cloud 22 can also control the operation of the edge terminal 20 by sending instructions to the edge terminal 20 according to the obtained inference results.

[0016] The edge terminal 20 of this embodiment performs object detection, which is a predetermined inference process, on video as described above, and includes a video transmission module 24, an example of which is shown in Fig. 2. The video transmission module 24 shown in Fig. 2 includes a camera 30, an image encoding device 32, a storage 34, and a wireless module 36.

[0017] The camera 30 is an example of a device for acquiring video in the edge terminal 20. The camera 30 outputs the video obtained by capturing the video to the image encoding device 32. Note that, in this embodiment, a form in which the image encoding device 32 processes the video obtained by the camera 30 will be described, but the images to be processed by the image encoding device 32 are not limited to the video obtained by the camera 30. For example, the images may be moving images such as animations composed of multiple images drawn by the user in the edge terminal 20. In this embodiment, the video input from the camera 30 to the image encoding device 32 is an example of a moving image of the present disclosure.

[0018] The image encoding device 32 is a device, such as an accelerator, for performing image encoding and predetermined inference processing on the video input from the camera 30. The image encoding device 32 outputs detection results obtained by object detection and bit streams obtained by image encoding to the storage 34. The wireless module 36 reads the bit stream stored in the storage 34 and transmits it to the cloud 22 when an event is detected, based on instructions from a control unit 44 (see FIG. 3 ) provided in the image encoding device 32.

[0019] The storage 34 stores the bit stream and detection results obtained by the image encoding device 32. The storage 34 also stores an image encoding program executed by the image encoding device 32. The image encoding program may be a single program, or a group of programs configured by multiple programs or modules. The storage 34 is configured by a DRAM, HDD, SSD, or the like. The image encoding program of this embodiment is an example of a program of the present disclosure.

[0020] <Configuration of Image Encoding Device 32 According to This Embodiment> FIG. 3 is a block diagram showing the hardware configuration of the image encoding device 32 according to this embodiment.

[0021] 3, the image encoding device 32 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a memory interface (I / F) 14, an input unit 15, and an external interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0022] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads the programs from the storage 34 and executes the programs using the RAM 13 as a work area. The CPU 11 controls each of the above components and performs various arithmetic processing in accordance with the programs stored in the storage 34.

[0023] The ROM 12 stores various programs and various data. The RAM 13 serves as a working area for temporarily storing programs or data.

[0024] The input unit 15 is used to input various types of information including images captured by the camera 30 .

[0025] The external interface 17 is an interface for reading out the bit stream generated by the encoding unit 42 (described later) and the detection result obtained by the detection unit 46 from the storage 34 and sending them to the outside.

[0026] Next, a description will be given of the functional configuration of the image encoding device 32. Fig. 4 is a block diagram showing an example of the functional configuration of the image encoding device 32.

[0027] As shown in FIG. 4, the image encoding device 32 functionally comprises a pre-processing unit 40, an encoding unit 42, a control unit 44, and a detection unit 46.

[0028] The pre-processing unit 40 converts the video input from the camera 30 into a format (here, YUV420) that can be processed by the encoding unit 42, and writes the converted video to the storage 34 via the memory interface 14. As an example, the pre-processing unit 40 of this embodiment converts the video input from the camera 30 into the YUV420 format. The pre-processing unit 40 also converts the video input from the camera 30 into a format that can be processed by the detection unit 46, and writes the converted video to the storage 34. As an example, the pre-processing unit 40 of this embodiment converts the video input from the camera 30 into the RGB format. In this embodiment, an image before being encoded by the encoding unit 42 is called an "original image."

[0029] The encoding unit 42 encodes each region, which is a coding block within a frame, using coding parameters. Specifically, the encoding unit 42 performs compression encoding of the input video using a lossy coding method. The encoding unit 42 of this embodiment is a so-called encoder. The encoding unit 42 reads the video written to the storage 34, performs compression encoding in accordance with a video coding standard such as H.264 / AVC (Advanced Video Coding), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding), and writes the resulting bitstream to the storage 34. The encoding unit 42 performs compression encoding of the image in units of regions, which are coding blocks, in accordance with the coding parameters determined by the control unit 44. An example of an area that is a coding block is a coding tree unit (CTU) or a coding unit (CU).

[0030] The detection unit 46 detects objects in each frame constituting a moving image. Specifically, the detection unit 46 performs a predetermined object detection process on the original image and outputs the detection result to the storage 34. The detection unit 46 of this embodiment is an AI (Artificial Intelligence) inference core capable of executing an object detection model previously trained by a neural network, and is capable of outputting the position and size of a specific object, such as a person or a vehicle, previously specified by a user as a Bbox (Bounding Box). The processes that the detection unit 46 can perform are not limited to object detection. For example, the detection unit 46 can also perform predetermined inference processes such as tracking of specific objects, segmentation of each object, and generation of an importance map. In cases where the detection unit 46 does not directly output the position and size of the specific object, such as when the detection result output by the detection unit 46 is an importance map, the position and size of the specific object may be calculated by performing post-processing (not shown). For example, the position of a group of pixels whose importance indicated by the importance map is equal to or greater than a predetermined threshold value, or the number of pixels, can be regarded as the position and size of a specific object.

[0031] The control unit 44 determines coding parameters for each region, which is a coding block in the coding unit 42, in accordance with a predetermined target code amount. In this embodiment, the control unit 44 uses a quantization parameter (hereinafter referred to as QP (Quantize Parameter)) as an example of a coding parameter, but the coding parameter is not limited to this. In general, there is a strong correlation between the target code amount and the coding parameter QP. The smaller the target code amount, the higher the coding parameter QP is set, and the larger the target code amount, the lower the coding parameter QP is set. Therefore, it is possible to use the target code amount as is as the coding parameter. The control unit 44 can dynamically change the coding parameter QP for each region based on multiple detection results output by the detection unit 46.

[0032] Each of the preprocessing unit 40, the encoding unit 42, and the detection unit 46 may be realized by a dedicated accelerator.

[0033] The detection unit 46 performs object detection on the video processed by the pre-processing unit 40, and outputs a plurality of B-boxes as the detection results.

[0034] As shown in FIG. 5, the control unit 44 includes a determination unit 50, a parameter determination unit 52, and a correction unit .

[0035] The determination unit 50 determines whether each region in each frame is a region of interest using the detection result of the detection unit 46. Specifically, the determination unit 50 uses the detection result output from the detection unit 46 to determine whether each region that is an encoding block is a region of interest (e.g., ROI).

[0036] More specifically, the determination unit 50 determines whether each region in each frame is a region of interest based on whether at least a portion of an object detected by the detection unit 46 is included in the region, or whether the distance to the object is equal to or less than a threshold. In this embodiment, the determination unit 50 determines regions that overlap with and are close to the B box obtained as the object detection result as regions of interest, and determines other regions as regions of non-interest. Note that in this embodiment, a region that is a coding block (e.g., a CTU) is used as a single unit to determine a region of interest, but a plurality of coding blocks may be used as a single unit to determine a region of interest, or it is also possible to determine a region of interest in smaller units (e.g., coding units (CUs) or transform units (TUs)).

[0037] The parameter determination unit 52 determines coding parameters for controlling the image quality of each region within a frame. Specifically, the parameter determination unit 52 determines a reference coding parameter BaseQP for each frame or each region, which is each coding block, based on a target code amount predetermined by a user and the code amount obtained from the coding unit 42.

[0038] The modifying unit 54 modifies the encoding parameters for a region determined to be a region of interest out of the regions in the frame, based on the determined encoding parameters and the size of the object corresponding to that region.

[0039] Specifically, the modification unit 54 determines DeltaQP, which is a modification amount of the encoding parameter, for each region using the output result of the determination unit 50, and determines the final encoding parameter QP to be applied to the region using the reference encoding parameter BaseQP and the modification amount DeltaQP. In this embodiment, the modification amount DeltaQP is an integer equal to or less than 0, and the encoding parameter QP to be applied to the region is calculated by QP = BaseQP + DeltaQP. Note that the method for determining the encoding parameter QP is not limited to this method. For example, if the modification amount DeltaQP is a weighting coefficient between 0 and 1, the encoding parameter QP to be applied to the region can also be calculated by QP = BaseQP * DeltaQP.

[0040] More specifically, for each region in the frame that has been determined to be a region of interest, the modification unit 54 determines the size of an object at least partially contained in the region or the size of an object whose distance from the region is equal to or less than a threshold, and determines the modification amount of the encoding parameter based on the determined size of the object. Also, for each region in the frame that has been determined to be a region of interest, the modification unit 54 determines the overlap rate between the region and a region representing the object, and determines the modification amount of the encoding parameter based on the size and overlap rate of the object.

[0041] An overview of a method for setting the correction amount DeltaQP will be described with reference to Figures 6A to 6D. Figures 6A to 6D show the relationship between the encoding parameter QP used during compression encoding in the encoding unit 42 and the inference accuracy for decoded video after encoding using the encoding parameter QP, for each size of detected object. Note that in this embodiment, AP (Average Precision), a general evaluation index for object detection accuracy, is used as the inference accuracy. Furthermore, the encoding parameter QP on the horizontal axis is the encoding parameter QP (QP for an I picture) that serves as a reference for the entire video, and the encoding parameter QP may change on a frame-by-frame basis depending on the GOP structure, the size of the Temporal ID of P pictures and B pictures, etc.

[0042] Figure 6A plots the inference processing accuracy AP for decoded video when all frames (images) of a specified video are compression-encoded using the encoding parameter QP shown on the horizontal axis. Figures 6B to 6D, on the other hand, show the degree of degradation in inference accuracy when compression-encoding is performed, with the inference accuracy when all frames of the specified video are not compression-encoded set as the reference (0%), with a larger negative value indicating a greater degradation in accuracy compared to the reference. The degree of degradation in inference accuracy is expressed by the following formula:

[0043] Degree of degradation of inference accuracy = -(100 - (inference accuracy for decoded video / inference accuracy without compression encoding) * 100)

[0044] As can be seen from Figure 6B, at a high encoding parameter QP of 42 or greater, the smaller the object size, the greater the degree of degradation in inference accuracy. Thus, in a low-bitrate environment where encoding is likely to be performed with a high encoding parameter QP, if the encoding parameter QP is controlled uniformly regardless of object size as in conventional technology, the inference accuracy of small objects may be significantly reduced compared to when no compression encoding is performed, compared to when compression encoding is not performed. This is particularly problematic in use cases where AI inference processing is required for a large number of small objects, such as surveillance using drones or cameras installed at high altitudes.

[0045] FIG. 6C illustrates an example in which the coding parameter QP for an area containing or adjacent to a small object is set to a correction amount DeltaQP smaller than the other object sizes, based on the tendency that the higher the coding parameter QP, the more rapidly the inference accuracy of smaller objects decreases, and the degree of decrease in inference accuracy diverges depending on the size of other objects. The starting point of the arrow indicates the coding parameter QP before application of the correction amount DeltaQP, and the end point of the arrow indicates the coding parameter QP after application of the correction amount DeltaQP. The dotted line on the horizontal axis indicates the coding parameter QP after application of the correction amount DeltaQP for each object size, and the dotted line on the vertical axis indicates the estimated change in inference accuracy after application of the correction amount DeltaQP. It can be seen that adaptively controlling the coding parameter QP according to the size of the object in this way can suppress the variation in the degree of decrease in inference accuracy for each object size. On the other hand, FIG. 6D illustrates an example in which a larger correction amount DeltaQP can be set than the example shown in FIG. 6C. In this case, since it is expected that the degree of degradation in inference accuracy after application of the correction amount DeltaQP will be approximately 0% regardless of the object size, the correction amount DeltaQP does not necessarily need to be changed according to the object size. The value of the correction amount DeltaQP can be specified in advance by the user depending on the use case or application using a decision table described below. When the correction amount DeltaQP is set to a large value as shown in FIG. 6D , the image quality for the region of interest improves and the degree of degradation in inference accuracy can be significantly suppressed regardless of the object size. However, the amount of code for the region corresponding to the region of interest increases, which may significantly degrade the image quality of non-interest regions. Therefore, in cases where a person wants to check the decoded video for purposes such as surveillance monitoring, the visibility of the video for non-interest regions is significantly reduced. On the other hand, when the correction amount DeltaQP is set to a small value as shown in FIG. 6C , the degree of degradation in inference accuracy in the region of interest is moderately suppressed, while the image quality of the non-interest regions is not impaired as much as in the example shown in FIG. 6D .

[0046] 7 is a diagram illustrating a method for determining an area of ​​interest performed by the determination unit 50. In this embodiment, the determination unit 50 determines an area as an area of ​​interest if a portion of a B box output by the detection unit 46 is included in the area, or if the area is close to any B box. Whether or not even a portion of a B box is included in the area is determined by determining that at least a portion of the B box is included in the area if the coordinates of the four corners of the area are (a0, a1, a2, a3) clockwise from the upper left coordinate, the upper left coordinate of the B box is (bx, by), the width is w, and the height is h.

[0047] bx<=an's x coordinate<=(bx+w-1) and by<=an's y coordinate<=(by+h-1) (n=0, 1, 2, 3)

[0048] On the other hand, whether Bbox is close to the region is determined by using a predetermined threshold value Th_margin for determining proximity (see the solid line with double arrows in Figure 7), and if the following equation holds for any n, it is determined that the Bbox is close to the region.

[0049] bx-Th_margin<=x coordinate of an<=(bx+w+Th_margin-1) and by-Th_margin<=y coordinate of an<=(by+h+Th_margin-1) (n=0, 1, 2, 3)

[0050] Furthermore, if it is determined that at least a portion of Bbox is included in a region, it is also possible to calculate the overlap ratio between the Bbox and the region. When Actu is the area of ​​the region and Abbox is the area of ​​Bbox, the overlap ratio D can be calculated according to the following formula:

[0051] D=(Actu∩Abbox) / min(Actu, Abbox)

[0052] <Operation of Image Encoding Apparatus According to This Embodiment> Next, the operation of the image encoding apparatus 32 according to this embodiment will be described.

[0053] 8 is a flowchart showing the flow of the image encoding process by the image encoding device 32. The image encoding process is performed by the CPU 11 reading out the image encoding program from the storage 34, loading it into the RAM 13, and executing it. It is also assumed that video is input to the image encoding device 32 from the camera 30. The image encoding process is an example of an image encoding method.

[0054] In step S100 , the CPU 11 functions as the pre-processing unit 40 to convert the video input from the camera 30 into a format that can be processed by the encoding unit 42 , and writes the converted video to the storage 34 via the memory interface 14 .

[0055] The processing of steps S102 to S112 is repeatedly executed for each frame constituting the video as the target frame.

[0056] In step S102 , the CPU 11 functions as the detection unit 46 to detect an object in the target frame of the video converted by the pre-processing unit 40 .

[0057] In step S104, the CPU 11 functions as the encoding unit 42 to calculate the amount of code for each area that is an encoding block in the target frame.

[0058] In step S106, the CPU 11 functions as the determination unit 50 and uses the detection result of the detection unit 46 to determine whether each area in the target frame is a region of interest.

[0059] In step S108, the CPU 11 functions as the parameter determination unit 52 to determine a reference coding parameter BaseQP for each region in the target frame from the code amount and the target code amount.

[0060] In step S110, the CPU 11, as the modification unit 54, modifies the reference encoding parameter BaseQP for each area in the target frame that is determined to be a region of interest, based on the determined encoding parameters and the size of the object corresponding to the area, thereby modifying the encoding parameter QP.

[0061] In step S112, the CPU 11, as the encoding unit 42, performs compression encoding of the image on a region-by-region basis in accordance with the encoding parameter QP determined by the control unit 44, writes the resulting bitstream to the storage 34, and terminates the image encoding process.

[0062] FIG. 9 is a flowchart of the attention area determination process for realizing step S106, which is executed for each area that is an encoding block such as a CTU.

[0063] First, in step S120, the CPU 11 functions as the determination unit 50 and acquires coordinate information of the area to be processed.

[0064] In step S122, the CPU 11, functioning as the determination unit 50, determines whether or not the search for all B boxes has been completed for the area to be processed. If the search for all B boxes has been completed, the process for that area ends and the process moves on to the next area. If the search for all B boxes has not been completed, in step S124, the CPU 11, functioning as the determination unit 50, acquires the coordinate information of the B box included in the detection result output by the detection unit 46.

[0065] In step S126, the CPU 11, functioning as the determination unit 50, determines whether or not at least a portion of each B box is included in the area to be processed using the method described above, etc. If at least a portion of the B box is included in the area to be processed, the process proceeds to step S128, and the CPU 11, functioning as the determination unit 50, stores information about the B box as the B box included in the area in question in the attention area determination result.

[0066] In step S130, the CPU 11 functions as the determination unit 50 and calculates the overlapping ratio between the Bbox and the area to be processed using the method described above or the like.

[0067] On the other hand, if at least a part of B box is not included in the region to be processed, the process proceeds to step S132, where the CPU 11, functioning as the determination unit 50, determines whether the B box is close to the region. If the B box is close to the region, the process proceeds to step S134, where the CPU 11, functioning as the determination unit 50, stores information about the B box as the B box close to the region in the attention region determination result.

[0068] 10A to 10D are diagrams showing an example of an attention area determination result generated by the determination unit 50. For each region constituting the input image, information on the B box contained in that region and the B boxes adjacent to that region is stored. As shown in (1) and (2), there are cases where the same B box is contained in different regions, and as shown in (2) and (3), there are also cases where multiple B boxes are contained in the same region. FIGS. 10B to 10D show an example of B box information stored in the attention area determination result. Because (1) and (2) are the same B box, the top left coordinate, width, height, and size of the B box are the same. However, the overlap rate between Bbox0 and region 0 is 80%, and the overlap rate between Bbox0 and region 1 is 20%. Therefore, even within the same B box, the information stored therein may differ in relation to each region.

[0069] 11 is a flowchart of the correction amount determination process for implementing step S110. The correction amount determination process is executed for each region.

[0070] First, in step S140, the CPU 11, functioning as the correction unit 54, refers to the attention area determination result generated by the determination unit 50, and acquires the stored B-box information.

[0071] In step S142, the CPU 11, functioning as the modifying unit 54, determines whether or not any Bbox information is stored for the region to be processed. If no Bbox information is stored for the region to be processed, the process proceeds to step S158, and the CPU 11, functioning as the modifying unit 54, determines the encoding parameter BaseQP determined by the parameter determining unit 52 as the encoding parameter QP to be used for encoding the region.

[0072] On the other hand, if B box information is stored for the region to be processed, the process proceeds to step S144, where the CPU 11, functioning as the correction unit 54, determines whether or not a B box is included in the region to be processed. If a B box is included in the region to be processed, the process proceeds to step S146, where the CPU 11, functioning as the correction unit 54, references the attention region determination result generated by the determination unit 50 and acquires the smallest size of the B boxes included in the region. Note that if only one B box is included in the region to be processed, the size of that B box can be used as is, and there is no need to acquire the smallest size. Furthermore, in this embodiment, the smallest size among multiple B boxes is acquired, but the method of acquiring the size is not limited to this, and it is sufficient to obtain a unique size from multiple B boxes. For example, the largest size among multiple B boxes may be acquired, or the average size of multiple objects may be acquired.

[0073] In step S148 , the CPU 11 functions as the modifying unit 54 to determine the modification amount DeltaQP from a determination table, which will be described later, based on the acquired object size and the encoding parameter BaseQP determined by the parameter determining unit 52 .

[0074] In step S150, the CPU 11, as the correction unit 54, adjusts the correction amount DeltaQP according to the overlap ratio between the B box included in the attention area determination result and the area in question. For example, in an area with an overlap ratio of 90% or more, it is considered that the image quality of the area in question significantly affects the detection accuracy of B box during inference processing of the compressed and encoded video. Therefore, by decreasing the correction amount DeltaQP of the area in question by -1, the correction amount DeltaQP is adjusted so that the image quality is relatively improved compared to areas with a lower overlap ratio. On the other hand, it is considered that an area with an overlap ratio of less than 10% has a relatively smaller impact on the detection accuracy of B box than areas with a larger overlap ratio. Therefore, by increasing the correction amount DeltaQP of the area in question by +1, the correction amount DeltaQP is adjusted so as to reduce the amount of generated code, thereby prioritizing the image quality of areas with a larger overlap ratio.

[0075] In step S156, the CPU 11, functioning as the modifying unit 54, determines the coding parameter QP to be used for coding the region based on the coding parameter BaseQP determined by the parameter determining unit 52 and the modification amount DeltaQP.

[0076] On the other hand, if there is no B box included in the area to be processed, the process proceeds to step S152, and the CPU 11, functioning as the correction unit 54, acquires the smallest size among the B boxes adjacent to the area to be processed. Note that if there is only one B box adjacent to the area to be processed, the size of that B box can be used as is, and there is no need to acquire the smallest size. Furthermore, in this embodiment, the smallest size among multiple B boxes is acquired, but the method of acquiring the size is not limited to this. For example, the largest size among multiple B boxes may be acquired, or the average size of multiple objects may be acquired.

[0077] In step S154 , the CPU 11 functions as the modifying unit 54 to determine the modification amount DeltaQP from a determination table, which will be described later, based on the acquired object size and the encoding parameter BaseQP determined by the parameter determining unit 52 .

[0078] Here, it is considered that an area that does not include at least a part of B box has a relatively smaller influence on the detection accuracy of B box than an area that includes B box.

[0079] Therefore, in this embodiment, in step S155, the CPU 11 functions as the modifying unit 54 to adjust the modification amount DeltaQP of the region by +1 to reduce the amount of generated code, thereby prioritizing the image quality of the region including the B box. Then, the CPU 11 proceeds to step S156.

[0080] In this embodiment, the correction amount DeltaQP determined by the determination table is further adjusted based on the overlap ratio between the region and the B box and whether or not the region includes a B box, but the method for adjusting the correction amount DeltaQP is not limited to this. For example, the determination unit 50 can also refer to the number of pieces of B box information for each region included in the attention region determination result and adjust the correction amount DeltaQP in accordance with the number of B boxes included in or adjacent to the region.

[0081] In addition, if the value of the correction amount DeltaQP becomes greater than 0 due to adjustment of the correction amount DeltaQP as described above, there is a risk that the image quality will be lower than that of non-interest areas, so the correction unit 54 may perform clipping processing to ensure that the value of the correction amount DeltaQP is always less than 0.

[0082] 12 is a diagram showing an example of a decision table referenced by the modification unit 54, and assumes an example of setting the modification amount DeltaQP for the graph shown in FIG. 6D. In the graph shown in FIG. 6D, when the encoding parameter QP is 42 or greater, the smaller the object, the greater the degree of degradation in inference accuracy tends to be. Taking this into consideration, the modification amount DeltaQP is set so that the higher the encoding parameter BaseQP, the better the image quality of areas containing or adjacent to smaller objects. Specifically, when the object size is equal to or less than the threshold Th_small, and the encoding parameter BaseQP is greater than the threshold Th_qp0 but less than or equal to the threshold Th_qp1, the value of the modification amount DeltaQP is set to -2. Furthermore, when the object size is equal to or less than the threshold Th_small, and the encoding parameter BaseQP is greater than the threshold Th_qp1, the value of the modification amount DeltaQP is set to -3.

[0083] Furthermore, when the object size is greater than the threshold Th_small and equal to or less than the threshold Th_large, and the coding parameter BaseQP is greater than the threshold Th_qp0 and equal to or less than the threshold Th_qp1, the value of the modification amount DeltaQP is set to −1. Furthermore, when the object size is equal to or less than the threshold Th_small, and the coding parameter BaseQP is greater than the threshold Th_qp1, the value of the modification amount DeltaQP is set to −2.

[0084] Furthermore, when the object size is larger than the threshold Th_large and the coding parameter BaseQP is larger than the threshold Th_qp0 and equal to or smaller than the threshold Th_qp1, the value of the correction amount DeltaQP is set to 0. Furthermore, when the object size is equal to or smaller than the threshold Th_small and the coding parameter BaseQP is larger than the threshold Th_qp1, the value of the correction amount DeltaQP is set to −1.

[0085] On the other hand, when the encoding parameter QP is less than 42, there is no significant difference in the degree of degradation of inference accuracy for each object size, and the value fluctuates at a value equivalent to the inference accuracy (0%) when no compression encoding is performed. Therefore, when the encoding parameter BaseQP is equal to or less than the threshold Th_qp0, the value of the modification amount DeltaQP is set to 0, and image quality control for the region of interest is not performed. Note that, although only one type of decision table is used in this embodiment, multiple types of decision tables can also be used. For example, the modification amount DeltaQP may be determined using two types of decision tables: a decision table for the region including Bbox and a decision table for the region adjacent to Bbox.

[0086] As described above, the image coding device according to this embodiment modifies coding parameters for regions determined to be regions of interest within frames constituting a video based on the size of the object corresponding to the region, and codes each region within the frame using the coding parameters. This makes it possible to compress data while suppressing a decrease in inference accuracy according to the size of the object.

[0087] Furthermore, the present invention is not limited to the device configuration and operation of the above-described embodiment, and various modifications and applications are possible within the scope of the gist of the present invention.

[0088] For example, the various processes executed by the CPU after reading software (programs) in the above embodiments may be executed by various processors other than the CPU. Examples of processors in this case include dedicated electrical circuits, such as programmable logic devices (PLDs) (such as field-programmable gate arrays (FPGAs)) whose circuit configuration can be changed after manufacture, and application-specific integrated circuits (ASICs) that are processors having circuit configurations designed specifically for executing specific processes. Furthermore, the image encoding process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, etc.). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

[0089] In addition, in each of the above embodiments, the image encoding program is described as being pre-stored (installed) in the storage 34, but the present invention is not limited to this. The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

[0090] The following additional notes are provided regarding the above-described embodiments.

[0091] (Supplementary Item 1) An image coding device comprising: a memory; and at least one processor connected to the memory, wherein the processor is configured to: detect an object in each frame constituting a moving image; determine whether each region in each frame is an area of ​​interest using the detection results; determine coding parameters for controlling image quality for each region in the frame; modify the coding parameters for each region in the frame that is determined to be an area of ​​interest based on the size of the object corresponding to the region; and code each region in the frame using the coding parameters.

[0092] (Supplementary Item 2) A non-transitory storage medium storing a program executable by a computer to perform an image encoding process, wherein the image encoding process is configured to: detect an object in each frame constituting a moving image; determine whether each area in each frame is an area of ​​interest using the detection results; determine encoding parameters for controlling image quality for each area in the frame; modify the encoding parameters for each area in the frame that is determined to be an area of ​​interest based on the size of the object corresponding to the area; and encode each area in the frame using the encoding parameters.

[0093] (Supplementary Item 3) An image coding device including: a detection unit that detects an object in each frame that constitutes a moving image; a determination unit that determines a region of interest in each frame using the detection result of the detection unit; a coding unit that performs coding using coding parameters for controlling image quality for each region in the frame; and a modification unit that determines a size of an object corresponding to the region of interest, and modifies the coding parameters based on modification conditions that include the size of the object and an image quality index obtained from coding parameters that have been previously assigned to the region of interest.

[0094] (Supplementary Item 4) The image encoding device described in Supplementary Item 3 is characterized in that the determination unit determines whether the object detected by the detection unit is included in or close to a region of interest, and if the object detected by the detection unit is included in the region of interest, the object is considered to belong to the region of interest, and if the object detected by the detection means is not included in the region of interest, an object close to the region of interest is considered to belong to the region of interest.

[0095] (Supplementary Item 5) The image encoding device described in Supplementary Item 3 or 4 is characterized in that, when there are multiple objects belonging to the area of ​​interest, the correction unit determines the size of the object corresponding to the area of ​​interest to be either the size of the smallest object among the multiple objects, the size of the largest object, or the average size of the multiple objects.

[0096] (Supplementary Item 6) The image encoding device described in any one of Supplementary Items 3 to 5, characterized in that the correction unit determines the overlap ratio of an object corresponding to the region of interest based on the overlap ratio of one or more objects belonging to the region of interest with the region of interest, and the correction condition further includes the determined overlap ratio of the objects.

[0097] (Supplementary Item 7) The image encoding device according to any one of Supplementary Items 3 to 6, wherein the correction conditions further include the number of objects belonging to the region of interest.

[0098] (Appendix 8) An image encoding device described in any one of appendixes 3 to 7, further comprising a decision table that holds a predetermined amount of modification of the encoding parameters based on the modification conditions, and the modification of the encoding parameters by the modification unit is realized by the modification unit obtaining the amount of modification from the decision table and modifying the encoding parameters based on the amount of modification.

[0099] (Appendix 9) An image coding device described in any one of appendixes 3 to 8, characterized in that the image quality index is a quantization parameter, and the modification unit modifies the encoding parameters so that the image quality of the region of interest becomes relatively higher the larger the value of the quantization parameter and the smaller the size of the object determined by the modification unit.

[0100] (Appendix 10) An image coding device described in any one of appendixes 3 to 9, characterized in that the image quality index is a target code size, and the modification unit modifies the coding parameters so that the image quality of the area of ​​interest becomes relatively higher the smaller the value of the target code size and the smaller the size of the determined object.

[0101] (Supplementary Item 11) The image coding device according to Supplementary Item 4, wherein the correction unit corrects the coding parameters for a region of interest that includes an object so that the image quality is relatively higher than that of a region of interest that only has nearby objects.

[0102] (Supplementary Item 12) The image encoding device according to Supplementary Item 6, wherein the correction unit corrects the encoding parameters so that the image quality of the region of interest becomes relatively higher as the overlap ratio of the object corresponding to the region of interest becomes larger.

[0103] (Supplementary Item 13) The image encoding device according to Supplementary Item 7, wherein the modification unit modifies the encoding parameters so that the image quality of the region of interest becomes relatively higher as the number of objects belonging to the region of interest increases.

[0104] REFERENCE SIGNS LIST 11 CPU 13 RAM 20 Edge terminal 24 Video transmission module 30 Camera 32 Image encoding device 34 Storage 40 Preprocessing unit 42 Encoding unit 44 Control unit 46 Detection unit 50 Determination unit 52 Parameter determination unit 54 Correction unit 100 Event-driven inference system

Claims

1. An image coding device including: a detection unit that detects objects in each frame that constitutes a moving image; a determination unit that uses the detection results of the detection unit to determine whether each area in each frame is an area of ​​interest; a parameter determination unit that determines coding parameters for controlling the image quality of each area in the frame; a modification unit that modifies the coding parameters for each area in the frame that is determined to be an area of ​​interest based on the size of the object corresponding to the area; and a coding unit that encodes each area in the frame using the coding parameters.

2. The image coding device of claim 1, wherein the determination unit determines whether each region in each frame is a region of interest based on whether at least a portion of an object detected by the detection unit is included in the region or whether the distance to the object is less than a threshold, and the modification unit determines, for each region in the frame determined to be a region of interest, the size of the object at least a portion of which is included in the region or the size of the object whose distance to the region is less than a threshold, and modifies the coding parameters based on the determined size of the object.

3. The image encoding device of claim 1, wherein the modification unit determines the overlap ratio between each region in the frame that is determined to be the region of interest and the region representing the object, and modifies the encoding parameters based on the size of the object and the overlap ratio.

4. An image coding method executed by a computer, which includes the steps of: detecting an object in each frame constituting a moving image; using the detection results to determine whether each area in each frame is an area of ​​interest; determining coding parameters for controlling the image quality of each area in the frame; modifying the coding parameters for an area in the frame that is determined to be an area of ​​interest based on the size of the object corresponding to the area; and coding each area in the frame using the coding parameters.

Images