Image processing system, image processing apparatus, image processing method, and image processing program

Abstract

Provided is an image processing system suited to transmitting an image to be used in AI-based recognition processing. This image processing device includes: a determination unit that determines, on the basis of the results of recognition processing, a target region necessary for recognizing a recognition target in image data and a non-target region other than the target region, and a quantized value for the target region necessary for recognizing the recognition target and a quantized value for the non-target region; a first encoding unit that encodes all regions of the image data using the quantized value for the target region, and generates first encoded data; a second encoding unit that encodes all regions of the image data using the quantized value for the non-target region, and generates second encoded data; a reconfiguration unit that generates reconfigured image data by using the target region in first decoded data obtained by decoding the first encoded data, and the non-target region in second decoded data obtained by decoding the second encoded data; and a re-encoding unit that re-encodes the reconfigured image data and generates re-encoded data.

Description

Technical Field

[0001] The present invention relates to an image processing system, an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] Generally, when recording or transmitting image data, the data size is reduced by encoding to reduce the recording cost and the transmission cost.

[0003] On the other hand, when recording or transmitting image data for the purpose of being used in recognition processing by AI (Artificial Intelligence), the quantization value of each region (a parameter that determines the compression rate, including quantization parameters, quantization steps, etc. For example, it corresponds to the QP value of the moving image encoding standard H.265 / HEVC) is increased (that is, at the limit quantization value) for encoding.

[0004] Here, in the case of an imaging device with a specification constraint that different desired quantization values cannot be set for each region of the captured image (for example, the same quantization value is set for the entire region), the above encoding method cannot be applied.

[0005] On the contrary, for example, if processing such as black painting is performed on regions other than the target region necessary for recognizing the recognition target and then the entire region is encoded with the limit quantization value, even in the case of the above imaging device, the data size of the encoded data can be reduced.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

[0007] However, when processing such as blackening is performed on areas other than the target area, it becomes difficult to use the areas other than the target area in the decoded data as image data. Therefore, in order to make the areas other than the target area also available, for example, after performing processing such as blackening on the target area, the entire area is encoded with a predetermined quantization value and separately transmitted as encoded data. According to such a method, by reconstructing two types of decoded data in the receiving device, the AI can recognize the recognition target, and image data that can utilize areas other than the target area can be generated.

[0008] On the other hand, in the case of such a method, two types of encoded data are transmitted for one image, and the receiving device needs to incorporate a function of receiving and decoding the two types of encoded data and a function of reconstructing them.

[0009] One aspect aims to provide an image processing system, an image processing device, an image processing method, and an image processing program suitable for transmitting an image used for recognition processing by AI. [Means for Solving the Problems]

[0010] According to one aspect, the image processing system a determination unit that determines, based on the result of the recognition process, a target area and a non-target area outside the target area that are necessary to recognize the recognition target in the image data, and a quantization value of the target area and a quantization value of the non-target area that are necessary to recognize the recognition target; a first encoding unit that encodes the entire area of the image data with the quantization value of the target area to generate first encoded data; A second encoding unit that encodes the entire area of the image data with the quantization value of the non-target area to generate second encoded data; A reconstruction unit that generates reconstructed image data using the target area in the first decoded data obtained by decoding the first encoded data and the non-target area in the second decoded data obtained by decoding the second encoded data; And a re-encoding unit that re-encodes the reconstructed image data to generate re-encoded data.

Advantages of the Invention

[0011] It is possible to provide an image processing system, an image processing apparatus, an image processing method, and an image processing program suitable for transmitting an image used for recognition processing by AI.

Brief Description of the Drawings

[0012]

Figure 1A

Figure 1B

Figure 1C

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17A

Figure 17B

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Embodiments for Carrying Out the Invention

[0013] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] [First Embodiment] <System Configuration of Image Processing System> First, the system configuration of an image processing system that encodes and transmits moving image data, performs recognition processing of decoded data using AI at the transmission destination, records the encoded data, and displays the decoded data to the user as necessary will be described.

[0015] (1) System Configuration 1 FIG. 1A is a first diagram showing an example of the system configuration of an image processing system. As shown in FIG. 1A, the image processing system 100 includes an imaging device 110, a hierarchical encoding device 111, and a server device 130. The hierarchical encoding device 111 and the server device 130 are communicably connected via a network 140.

[0016] The imaging device 110 performs shooting at a predetermined frame period and transmits the moving image data to the hierarchical encoding device 111.

[0017] The hierarchical encoding device 111 is disposed in the vicinity of the imaging device 110. The hierarchical encoding device 111 encodes the image data of each frame included in the moving image data and generates first encoded data. When generating the first encoded data, the hierarchical encoding device 111 · determines a region (target region) necessary for AI to recognize the recognition target included in the image data, and · determines a limit quantization value (limit quantization value) that is the limit necessary for AI to recognize the recognition target included in the image data, and encodes the entire region of the image data with the same limit quantization value.

[0018] Further, the hierarchical encoding device 111 encodes the image data of each frame included in the moving image data and generates second encoded data. When generating the second encoded data, the hierarchical encoding device 111 · determines an area other than the area (non-target area) necessary for the AI to recognize the recognition target included in the image data, and · a predetermined quantization value suitable for encoding the non-target area, and encodes the entire area of the image data with the same predetermined quantization value.

[0019] Furthermore, the hierarchical encoding device 111 · information regarding the area and quantization value determined when generating the first encoded data and the first encoded data, and · information regarding the area and quantization value determined when generating the second encoded data and the second encoded data, and transmits the information to the server device 130.

[0020] An image processing program is installed in the server device 130. When the program is executed, the server device 130 functions as the transcoding unit 121. Also, an image recognition program is installed in the server device 130. When the program is executed, the server device 130 functions as the re-encoded data acquisition unit 131, the video analysis unit 132, and the video display unit 133.

[0021] The transcoding unit 121 decodes the first encoded data and the second encoded data transmitted from the hierarchical encoding device 111 and generates first decoded data and second decoded data. Also, the transcoding unit 121 extracts the target area from the first decoded data and the non-target area from the second decoded data based on the information regarding the area transmitted from the hierarchical encoding device 111. Further, the transcoding unit 121 generates reconstructed image data by combining the extracted target area and non-target area.

[0022] Also, the transcoding unit 121, based on the information regarding the area and quantization value transmitted from the hierarchical encoding device 111, ·Encode the target area in the reconstructed image data with a quantization value equal to or close to the limit quantization value, ·Encode the non-target area in the reconstructed image data with a predetermined quantization value, By doing this, re-encoded data is generated. Further, the transcoding unit 121 notifies the re-encoded data acquisition unit 131 of the re-encoded data.

[0023] The re-encoded data acquisition unit 131 acquires the re-encoded data, notifies the video analysis unit 132, and stores it in the re-encoded data storage unit 134.

[0024] The video analysis unit 132 decodes the re-encoded data notified by the re-encoded data acquisition unit 131 and generates decoded data. Also, the video analysis unit 132 performs recognition processing by AI on the generated decoded data to recognize the recognition target included in the decoded data. Further, the video analysis unit 132 outputs the recognition result to the user.

[0025] Also, the video display unit 133 reads and decodes the re-encoded data within the range specified by the user from the re-encoded data stored in the re-encoded data storage unit 134, and generates decoded data. Also, the video display unit 133 displays the generated decoded data as video data to the user.

[0026] In this way, in the case where the hierarchical encoding device 111 arranged in the vicinity of the imaging device 110 of the image processing system 100 cannot set different quantization values for each area with respect to the photographed image data, ·A new transcoding unit 121 is arranged, ·After integrating the first encoded data and the second encoded data transmitted from the hierarchical encoding device 111 to generate re-encoded data, notify the re-encoded data acquisition unit 131.

[0027] Thus, according to the image processing system 100, · Since the first encoded data and the second encoded data are no longer directly input to the re-encoding data acquisition unit 131, it is no longer necessary to incorporate into the image recognition program the functions of receiving two types of encoded data and reconstructing them. · Since the first encoded data and the second encoded data are transmitted, the reduction in the amount of transmission data between the hierarchical encoding device 111 and the server device 130 can be maintained. · Since re-encoded data having a data amount comparable to that of the first encoded data and the second encoded data is stored, the amount of stored data stored in the server device 130 can be reduced. · Since the target area necessary for the AI to recognize and the limit quantization value necessary for the AI to recognize are ensured, the video analysis unit 132 can realize the recognition process by the AI with high recognition accuracy. · When the re-encoded data stored in the re-encoded data storage unit 134 is read out and decoded, a non-target area, which is an area other than the target area in the decoded data, can be used as image data.

[0028] Thus, according to the first embodiment, it is possible to provide an image processing system 100, an image processing method, and an image processing program suitable for transmitting an image used for the recognition process by the AI.

[0029] (2) System Configuration 2 FIG. 1B and FIG. 1C are the second and third diagrams showing an example of the system configuration of the image processing system. As shown in FIG. 1B and FIG. 1C, the image processing system 100' or 100'' includes an imaging device 110, a hierarchical encoding device 111, an image processing device 120, and a server device 130. The image processing device 120 and the server device 130 (or the hierarchical encoding device 111 and the image processing device 120) are communicably connected via a network 140.

[0030] Among these, since the imaging device 110 and the hierarchical encoding device 111 are the same as the imaging device 110 and the hierarchical encoding device 111 described in FIG. 1A, the description thereof is omitted here. Note that the hierarchical encoding device 111 · Information regarding the first encoded data, the region determined when generating the first encoded data, and the quantization value, and · Information regarding the second encoded data, the region determined when generating the second encoded data, and the quantization value, and are transmitted to the image processing apparatus 120.

[0031] An image processing program is installed in the image processing apparatus 120, and when the program is executed, the image processing apparatus 120 functions as a transcoding unit 121.

[0032] The transcoding unit 121 decodes the first encoded data and the second encoded data transmitted from the hierarchical encoding apparatus 111, and generates first decoded data and second decoded data. Further, the transcoding unit 121 extracts a target region from the first decoded data and extracts a non-target region from the second decoded data based on the information regarding the region transmitted from the hierarchical encoding apparatus 111. Further, the transcoding unit 121 generates reconstructed image data by combining the extracted target region and non-target region.

[0033] Also, among the reconstructed image data generated by the transcoding unit 121 based on the information regarding the region and the quantization value transmitted from the hierarchical encoding apparatus 111, · The target region is encoded with a limit quantization value or a quantization value close to the limit quantization value, · The non-target region is encoded with a predetermined quantization value, thereby generating re-encoded data. Further, the transcoding unit 121 transmits the re-encoded data to the server apparatus 130.

[0034] An image recognition program is installed in the server apparatus 130, and when the image recognition program is executed, the server apparatus 130 functions as a re-encoded data acquisition unit 131, a video analysis unit 132, and a video display unit 133.

[0035] Note that the re-encoding data acquisition unit 131, video analysis unit 132, and video display unit 133 shown in FIGS. 1B and 1C are the same as the re-encoding data acquisition unit 131, video analysis unit 132, and video display unit 133 shown in FIG. 1A, so the description thereof is omitted here.

[0036] As described above, when the hierarchical encoding device 111 arranged near the imaging device 110 in the image processing system 100' or 100'' cannot set different quantization values for each region of the captured image data, · A new image processing device 120 is arranged and made to function as a transcoding unit 121. · After integrating the first encoded data and the second encoded data notified from the hierarchical encoding device 111 to generate re-encoded data, the re-encoded data is transmitted to the server device 130.

[0037] Accordingly, according to the image processing system 100' or 100'', · Since the first encoded data and the second encoded data are not directly input to the server device 130, it is not necessary to incorporate into the image recognition program of the server device 130 the functions of receiving and reconstructing the two types of encoded data. · Since re-encoded data having a data amount comparable to that of the first encoded data and the second encoded data is transmitted, it is possible to maintain a reduction in the amount of transmission data between the image processing device 120 and the server device 130 (or between the hierarchical encoding device 111 and the image processing device 120). · Since re-encoded data having a data amount comparable to that of the first encoded data and the second encoded data is stored, it is possible to reduce the amount of stored data stored in the server device 130. · Since the target region necessary for the AI to recognize and the limit quantization value necessary for the AI to recognize are ensured, the video analysis unit 132 can realize the recognition process by the AI with high recognition accuracy. · When the re-encoded data stored in the re-encoded data storage unit 134 is read out and decoded, a non-target region, which is a region other than the target region in the decoded data, can be used as image data.

[0038] As described above, according to the first embodiment, it is possible to provide an image processing apparatus 120, an image processing system 100' or 100'', an image processing method, and an image processing program suitable for transmitting an image used for recognition processing by AI.

[0039] <Hardware Configuration of Image Processing Apparatus and Server Apparatus> Next, the hardware configuration of the image processing apparatus 120 of the image processing system 100' or 100'', and the hardware configuration of the server apparatus 130 of the image processing system 100 or the server apparatus 130 of the image processing system 100' or 100'' will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the image processing apparatus and the server apparatus.

[0040] Among these, 2a in FIG. 2 is a diagram showing an example of the hardware configuration of the image processing apparatus 120 of the image processing system 100' or 100''. The image processing apparatus 120 includes a processor 201, a memory 202, an auxiliary storage device 203, an I / F (Interface) device 204, a communication device 205, and a drive device 206. Each hardware of the image processing apparatus 120 is interconnected via a bus 207.

[0041] The processor 201 includes various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 201 reads out and executes various programs (for example, an image processing program, etc.) on the memory 202.

[0042] The memory 202 includes main storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 201 and the memory 202 form a so-called computer, and the computer realizes various functions by the processor 201 executing various programs read out on the memory 202.

[0043] The auxiliary storage device 203 stores various programs and various data used when the various programs are executed by the processor 201.

[0044] The I / F device 204 is a connection device that connects the hierarchical encoding device 111, which is an example of an external device, and the image processing device 120.

[0045] The communication device 205 is a communication device for communicating with the server device 130 via a network.

[0046] The drive device 206 is a device for setting the recording medium 210. The recording medium 210 mentioned here includes media that optically, electrically, or magnetically record information, such as CD-ROMs, flexible disks, magneto-optical disks, etc. Further, the recording medium 210 may include semiconductor memories that electrically record information, such as ROMs, flash memories, etc.

[0047] Note that the various programs installed in the auxiliary storage device 203 are installed, for example, when the distributed recording medium 210 is set in the drive device 206 and the various programs recorded on the recording medium 210 are read out by the drive device 206. Alternatively, the various programs installed in the auxiliary storage device 203 may be installed by being downloaded from the network 140 via the communication device 205.

[0048] On the other hand, FIG. 2b is a diagram showing an example of the hardware configuration of the server device 130 of the image processing system 100 or the server device 130 of the image processing systems 100' or 100''. Since the hardware configuration of the server device 130 is generally the same as the hardware configuration of the image processing device 120 shown in FIG. 2a, here, the description will focus on the differences from the image processing device 120 shown in FIG. 2a.

[0049] The processor 221 reads and executes, for example, an image processing program, an image recognition program, etc. on the memory 222.

[0050] The I / F device 224 receives operations on the server device 130 via the operation device 231. Also, the I / F device 224 outputs the result of the processing by the server device 130 and displays it via the display device 232. Further, the communication device 225 communicates with the hierarchical encoding device 111 or the image processing device 120 via the network 140.

[0051] Hereinafter, for the sake of simplicity of explanation, details of the functional configurations of the respective devices in the case of the system configuration shown in FIG. 1A (details of the functional configuration of the hierarchical encoding device 111 of the image processing system 100, details of the functional configuration of the transcoding unit 121 of the server device 130, etc.) will be described.

[0052] <Functional Configuration of Hierarchical Encoding Device> First, the functional configuration of the hierarchical encoding device 111 of the image processing system 100 will be described with reference to FIG. 3. FIG. 3 is a first diagram showing an example of the functional configuration of the hierarchical encoding device.

[0053] A hierarchical encoding program is installed in the hierarchical encoding device 111, and when the program is executed, the hierarchical encoding device 111 functions as a compression information determination unit 310, a region separation unit 320, a first encoding unit 330, and a second encoding unit 340.

[0054] The compression information determination unit 310 is an example of a determination unit. The compression information determination unit 310 repeatedly performs encoding and decoding while changing the quantization value for the image data of each frame included in the moving image data, performs recognition processing by AI on each decoded data, and determines whether or not the recognition target can be recognized. Thereby, the compression information determination unit 310 determines the limit quantization value (limit quantization value) necessary for the AI to recognize the recognition target and determines the target region necessary for the AI to recognize the recognition target.

[0055] When the recognition target is included in the image data, the compression information determination unit 310 · Notify the determined target area and the non-target area derived from the determined target area to the area separation unit 320, and also notify them to the first encoding unit 330 and the second encoding unit 340 respectively. · Notify the determined limit quantization value to the first encoding unit 330, and notify a predetermined quantization value (a quantization value suitable for encoding the non-target area) to the second encoding unit 340.

[0056] Also, when the recognition target is not included in the image data, the compression information determination unit 310 · Notify the entire area to the area separation unit 320, and also notify it to the first encoding unit 330 and the second encoding unit 340. · Notify a predetermined quantization value to the first encoding unit 330 and the second encoding unit 340.

[0057] The area separation unit 320 separates the image data of each frame included in the moving image data based on the target area and the non-target area notified from the compression information determination unit 310. Specifically, the area separation unit 320 separates the image data of each frame included in the moving image data into · First image data composed of the image of the target area and the invalid image of the non-target area, · Second image data composed of the invalid image of the target area and the image of the non-target area. Here, the invalid image refers to an image in which the pixel value of each pixel is a predetermined pixel value (for example, a pixel value corresponding to black, etc.).

[0058] Also, among the separated image data, the area separation unit 320 · Notify the first image data composed of the image of the target area and the invalid image of the non-target area to the first encoding unit 330, · Notify the second image data composed of the invalid image of the target area and the image of the non-target area to the second encoding unit 340.

[0059] Note that when the entire area is notified from the compression information determination unit 310, the area separation unit 320 · Notify the first image data with the entire area being invalid image to the first encoding unit 330. · Notify the second image data consisting of an image of the entire area to the second encoding unit 340.

[0060] The first encoding unit 330 is an example of a first encoding section. It encodes the first image data notified from the area separation unit 320 using the limit quantization value (or a predetermined quantization value) notified from the compression information determination unit 310, and generates first encoded data. Further, the first encoding unit 330 includes information regarding the target area (or the entire area) and the limit quantization value (or a predetermined quantization value) notified from the compression information determination unit 310 in the generated first encoded data, and transmits it to the server device 130.

[0061] The second encoding unit 340 is an example of a second encoding section. It encodes the second image data notified from the area separation unit 320 using the predetermined quantization value notified from the compression information determination unit 310, and generates second encoded data. Further, the second encoding unit 340 includes information regarding the non-target area (or the entire area) and the predetermined quantization value notified from the compression information determination unit 310 in the generated second encoded data, and transmits it to the server device 130.

[0062] Note that the method by which the first encoding unit 330 includes information regarding the target area (or the entire area) and the limit quantization value (or a predetermined quantization value) in the first encoded data is arbitrary. Similarly, the method by which the second encoding unit 340 includes information regarding the non-target area (or the entire area) and the predetermined quantization value in the second encoded data is arbitrary.

[0063] As an example, there is a method of including the above information in a header that can be defined by a user in a packet, such as RTP (Real-time Transport Protocol), or in a part of the payload. Further, as another example, in the case where the encoding method is HEVC or the like, there is a method of including the above information in the NAL number that can be used by the user (the use is not determined by the standard).

[0064] <Functional Configuration of the Transcoding Unit> Next, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 will be described with reference to FIG. 4. FIG. 4 is a first diagram showing an example of the functional configuration of the transcoding unit.

[0065] As shown in FIG. 4, the transcoding unit 121 includes a first decoding unit 410, a second decoding unit 420, a reconstruction unit 430, a quantization value map generation unit 440, and a re-encoding unit 450.

[0066] The first decoding unit 410 receives the first encoded data (including information on regions and quantization values) transmitted from the hierarchical encoding device 111, and generates first decoded data by decoding the received first encoded data. Further, the first decoding unit 410 notifies the generated first decoded data, together with the information on regions and quantization values, to the reconstruction unit 430.

[0067] The second decoding unit 420 receives the second encoded data (including information on regions and quantization values) transmitted from the hierarchical encoding device 111, and generates second decoded data by decoding the received second encoded data. Further, the second decoding unit 420 notifies the generated second decoded data, together with the information on regions and quantization values, to the reconstruction unit 430.

[0068] The reconstruction unit 430 extracts an image of the target region from the first decoded data notified by the first decoding unit 410 based on the information on regions. Further, the reconstruction unit 430 extracts an image of the non-target region from the second decoded data notified by the second decoding unit 420 based on the information on regions. Further, the reconstruction unit 430 synthesizes the extracted image of the target region and the extracted image of the non-target region to generate reconstructed image data.

[0069] In addition, the reconstruction unit 430 notifies the generated reconstructed image data to the re-encoding unit 450. Furthermore, the reconstruction unit 430 · Information on the region and quantization value notified by the first decoding unit 410 (target region and limit quantization value), and · Information on the region and quantization value notified by the second decoding unit 420 (non-target region and predetermined quantization value), and are notified to the quantization value map generation unit 440.

[0070] The quantization value map generation unit 440 generates a quantization value map based on the information on the region and quantization value notified by the reconstruction unit 430. In the quantization value map generation unit 440, the quantization value map is generated by setting the limit quantization value or a quantization value close to the limit quantization value in the target region and setting a predetermined quantization value in the non-target region.

[0071] Also, the quantization value map generation unit 440 notifies the generated quantization value map to the re-encoding unit 450.

[0072] The re-encoding unit 450 is an example of a re-coding unit. Encoding processing is performed on the reconstructed image data notified by the reconstruction unit 430 using the quantization value map notified by the quantization value map generation unit 440 to generate re-coded data. Note that the re-encoding unit 450 is assumed to have a function of performing encoding processing using different quantization values for each region. Also, the re-encoding unit 450 notifies the generated re-coded data to the re-coded data acquisition unit 131.

[0073] In this way, in the transcoding unit 121, a quantization value map is generated based on the information on the region and quantization value determined in the hierarchical encoding device 111. Thereby, according to the image processing system 100 according to the first embodiment, equivalent image quality can be maintained before and after the transcoding unit 121.

[0074] Note that the encoding method used by the re-encoding unit 450 when performing the encoding process may be the same as or different from the encoding method used by the first encoding unit 330 and the second encoding unit 340 when performing the encoding process. For example, the encoding method used by the first encoding unit 330 and the second encoding unit 340 when performing the encoding process may be H.265 / HEVC, and the encoding method used by the re-encoding unit 450 when performing the encoding process may be H.264 / MPEG-4 AVC.

[0075] Also, the specifications of the re-encoding unit 450 may be the same as or different from the specifications of the first encoding unit 330 and the second encoding unit 340.

[0076] Note that when a plurality of encoding units (e.g., the first encoding unit 330, the second encoding unit 340) and a plurality of decoding units (e.g., the first decoding unit 410, the second decoding unit 420) are used, information regarding the region does not necessarily have to be transmitted and received.

[0077] For example, by using the information regarding the region transmitted and received between the other encoding unit and the decoding unit, the information regarding the region transmitted and received between one encoding unit and the decoding unit may be derived. In such a case, it is not necessarily required to transmit and receive the information regarding the region between the one encoding unit and the decoding unit.

[0078] <Specific Examples of the Processing of the Hierarchical Encoding Device and the Transcoding Unit> Next, specific examples of the processing of the hierarchical encoding device 111 and the transcoding unit 121 will be described. FIG. 5 is a first diagram showing specific examples of the processing of the hierarchical encoding device and the transcoding unit.

[0079] In FIG. 5, the image data 501 is image data for one frame included in the moving image data. As shown in FIG. 5, the region separation unit 320 divides the acquired image data 501 into · first image data composed of an image of the target region and an invalid image of the non-target region, and ·The second image data composed of the invalid image of the target area and the image of the non-target area, is separated into, ·The first encoding unit 330 encodes the first image data composed of the image of the target area and the invalid image of the non-target area using the limit quantization value to generate the first encoded data 502. ·The second encoding unit 340 encodes the second image data composed of the invalid image of the target area and the image of the non-target area using a predetermined quantization value to generate the second encoded data 512.

[0080] Also, as shown in FIG. 5, the first encoded data 502 generated by the first encoding unit 330 is transmitted to the transcoding unit 121 and decoded by the first decoding unit 410 to generate the first decoded data 503.

[0081] Similarly, the second encoded data 512 generated by the second encoding unit 340 is transmitted to the transcoding unit 121 and decoded by the second decoding unit 420 to generate the second decoded data 513.

[0082] Also, as shown in FIG. 5, the first decoded data 503 extracts the image of the target area by the reconstruction unit 430, and the second decoded data 513 extracts the image of the non-target area by the reconstruction unit 430. Also, the extracted image of the target area and the extracted image of the non-target area are combined by the reconstruction unit 430 to generate the reconstructed image data 520.

[0083] Also, as shown in FIG. 5, the generated reconstructed image data 520 is encoded by the re-encoding unit 450 using the quantization value map to generate the re-encoded data 530. The example in FIG. 5 shows the state where the re-encoding unit 450 encodes the reconstructed image data 520 with the limit quantization value for the target area and with the predetermined quantization value for the non-target area.

[0084] <Flow of image processing by the image processing system> Next, the flow of image processing by the image processing system 100 will be described. FIG. 6 is a first flowchart showing the flow of image processing.

[0085] In step S601, the imaging device 110 acquires moving image data.

[0086] In step S602, the hierarchical encoding device 111 determines a target area and a non-target area for the image data of each frame included in the moving image data.

[0087] In step S603, the hierarchical encoding device 111 determines a limit quantization value for the target area and a predetermined quantization value for the non-target area for the image data of each frame included in the moving image data.

[0088] In step S604, the hierarchical encoding device 111 generates first image data composed of the image of the target area and the invalid image of the non-target area, and second image data composed of the invalid image of the target area and the image of the non-target area.

[0089] In step S605, the hierarchical encoding device 111 encodes the first image data with the determined limit quantization value to generate first encoded data. Further, the hierarchical encoding device 111 includes information on the area and the quantization value in the generated first encoded data and transmits it to the server device 130.

[0090] In step S606, the hierarchical encoding device 111 encodes the second image data with the determined predetermined quantization value to generate second encoded data. Further, the hierarchical encoding device 111 includes information on the area and the quantization value in the generated second encoded data and transmits it to the server device 130.

[0091] In step S607, the transcoding unit 121 of the server device 130 decodes the first encoded data to generate first decoded data.

[0092] In step S608, the transcoding unit 121 of the server device 130 decodes the second encoded data and generates second decoded data.

[0093] In step S609, the transcoding unit 121 of the server device 130 synthesizes the image of the target area of the first decoded data and the image of the non-target area of the second decoded data to generate reconstructed image data.

[0094] In step S610, the transcoding unit 121 of the server device 130 generates a quantization value map with different quantization values for the target area and the non-target area based on the information on the area and the quantization value included in the first encoded data and the second encoded data.

[0095] In step S611, the transcoding unit 121 of the server device 130 re-encodes the reconstructed image data using the quantization value map and generates re-encoded data.

[0096] In step S612, the imaging device 110 determines whether to end the image processing. If it is determined in step S612 that the image processing is not to be ended (if NO in step S612), the process returns to step S601.

[0097] On the other hand, if it is determined in step S612 that the image processing is to be ended (if YES in step S612), the image processing is ended.

[0098] As is clear from the above description, the image processing system 100 according to the first embodiment has a transcoding unit 121, integrates the first encoded data and the second encoded data transmitted from the hierarchical encoding device 111, and generates re-encoded data.

[0099] Accordingly, according to the image processing system 100 according to the first embodiment, the first encoded data and the second encoded data are no longer directly input to the re-encoding data acquisition unit 131. As a result, according to the image processing system 100 according to the first embodiment, there is no need to incorporate into the image recognition program of the server device 130 the functions of receiving two types of encoded data and reconstructing them.

[0100] That is, according to the first embodiment, it is possible to provide an image processing system, an image processing method, and an image processing program suitable for transmitting an image used for recognition processing by AI in a server device.

[0101] [Second Embodiment] In the above first embodiment, the quantization value map generation unit 440 has been described as generating a quantization value map based on the information regarding the region and the quantization value notified from the reconstruction unit 430. However, the method for generating the quantization value map by the quantization value map generation unit 440 is not limited to this. For example, among the images of the non-target region, the invalid region that is not effective for display by the video display unit 133 may be re-encoded with the maximum quantization value. Alternatively, among the images of the non-target region, the invalid region that is not effective for display by the video display unit 133 may be made an invalid image in the reconstruction unit 430 and then re-encoded with an arbitrary quantization value. Hereinafter, the second embodiment will be described centering on the differences from the first embodiment.

[0102] <Functional Configuration of the Transcoding Unit> First, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the second embodiment will be described with reference to FIG. 7. FIG. 7 is a second diagram showing an example of the functional configuration of the transcoding unit.

[0103] The difference from FIG. 4 is that the functions of the reconstruction unit 710 and the quantization value map generation unit 720 are different from the functions of the reconstruction unit 430 and the quantization value map generation unit 440 shown in FIG. 4.

[0104] The reconstruction unit 710 extracts the image of the target area from the first decoded data notified by the first decoding unit 410 based on the information regarding the area. Also, the reconstruction unit 710 extracts the image of the non-target area from the second decoded data notified by the second decoding unit 420 based on the information regarding the area. Further, the reconstruction unit 710 synthesizes the extracted image of the target area and the extracted image of the non-target area to generate reconstructed image data.

[0105] Also, the reconstruction unit 710 notifies the re-encoding unit 450 of the generated reconstructed image data, and · information regarding the area and quantization value (target area and limit quantization value) notified by the first decoding unit 410, and · information regarding the area and quantization value (non-target area and predetermined quantization value) notified by the second decoding unit 420, to the quantization value map generation unit 720.

[0106] Furthermore, when an invalid area that is not effective for display by the video display unit 133 is specified in the image of the non-target area, the reconstruction unit 710 notifies the quantization value map generation unit 720 of the invalid area.

[0107] Alternatively, when an invalid area that is not effective for display by the video display unit 133 is specified in the image of the non-target area, the reconstruction unit 710 generates the reconstructed image data with the invalid area as an invalid image and notifies the re-encoding unit 450.

[0108] The quantization value map generation unit 720 generates a quantization value map based on the information regarding the area and quantization value notified by the reconstruction unit 710. In the quantization value map generation unit 720, the quantization value map is generated by setting the limit quantization value or a quantization value close to the limit quantization value for the target area and setting a predetermined quantization value for the non-target area.

[0109] Also, when the quantization value map generation unit 720 is notified of an invalid area from the reconstruction unit 710, it changes the quantization values of the notified invalid area in the generated quantization value map to the maximum quantization value.

[0110] Furthermore, the quantization value map generation unit 720 notifies the re-encoding unit 450 of the changed quantization value map.

[0111] Note that when the re-encoding unit 450 is notified of reconstructed image data with an invalid area as an invalid image from the reconstruction unit 710, it generates re-encoded data using the quantization value map generated by the quantization value map generation unit 720.

[0112] Also, when the re-encoding unit 450 is notified of reconstructed image data from the reconstruction unit 710, it generates re-encoded data using the changed quantization value map changed by the quantization value map generation unit 720.

[0113] In this way, by setting the maximum quantization value for the invalid area (or making the invalid area an invalid image), according to the image processing system 100 according to the second embodiment, · Re-encoded data with a reduced data amount compared to the first encoded data and the second encoded data will be stored, and the amount of stored data stored in the server device 130 can be further reduced.

[0114] <Specific Example of Processing of Hierarchical Encoding Device and Transcoding Unit> Next, a specific example of the processing of the hierarchical encoding device 111 and the transcoding unit 121 will be described. FIG. 8 is a second diagram showing a specific example of the processing of the hierarchical encoding device and the transcoding unit.

[0115] The difference from FIG. 5 is that when generating the re-encoded data 530, the invalid area 801 is specified, and the invalid area 801 is encoded with the maximum quantization value (or, after treating the invalid area 801 as an invalid image, encoded with an arbitrary quantization value). Another difference from FIG. 5 is that in the case of FIG. 8, the re-encoded data 810 is generated.

[0116] <Image processing flow by the image processing system> Next, the image processing flow by the image processing system 100 will be described. FIG. 9 is a second flowchart showing the image processing flow. The differences from FIG. 6 are steps S901 and S902.

[0117] In step S901, the transcoding unit 121 of the server device 130 identifies an invalid area that is not effective for the video display unit 133 to display among the images of the non-target area.

[0118] In step S902, the transcoding unit 121 of the server device 130 generates a quantization value map with different quantization values for the target area and the non-target area based on the information on the area and the quantization value included in the first encoded data and the second encoded data. Also, the transcoding unit 121 of the server device 130 changes the quantization value map so that the quantization value of the identified invalid area becomes the maximum quantization value. Alternatively, the transcoding unit 121 of the server device 130 generates reconstructed image data treating the identified invalid area as an invalid image.

[0119] As is clear from the above description, the image processing system 100 according to the second embodiment sets the quantization value of the invalid area to the maximum quantization value (or treats the invalid area as an invalid image). Thereby, according to the image processing system 100 according to the second embodiment, the amount of stored data stored in the server device 130 can be further reduced.

[0120] That is, according to the second embodiment, while enjoying the same effects as the first embodiment, the amount of stored data can be further reduced.

[0121] [Embodiment 3] In the above-described first and second embodiments, the case where the hierarchical encoding device 111 includes information on the region and quantization value in each of the first encoded data and the second encoded data and transmits it to the server device 130 has been described. However, the method of transmitting information on the region and quantization value is not limited to this. For example, it may be transmitted to the server device 130 separately from the first encoded data and the second encoded data. Hereinafter, the third embodiment will be described centering on the differences from the above-described first and second embodiments.

[0122] <Functional Configuration of Hierarchical Encoding Device> First, the functional configuration of the hierarchical encoding device 111 of the image processing system 100 according to the third embodiment will be described with reference to FIG. 10A. FIG. 10A is a second diagram showing an example of the functional configuration of the hierarchical encoding device.

[0123] The difference from FIG. 3 is that the functions of the compression information determination unit 1010, the first encoding unit 1020, and the second encoding unit 1030 are different from the functions of the compression information determination unit 310, the first encoding unit 330, and the second encoding unit 340 shown in FIG. 3.

[0124] The compression information determination unit 1010 repeatedly performs encoding and decoding on the image data of each frame included in the moving image data while changing the quantization value, performs recognition processing by AI on each decoded data, and determines whether or not the recognition target can be recognized. Thereby, the compression information determination unit 1010 determines the limit quantization value necessary for the AI to recognize the recognition target and determines the target region necessary for the AI to recognize the recognition target.

[0125] Also, when the recognition target is included in the image data, the compression information determination unit 1010 · Notifies the region separation unit 320 of the determined target region and the non-target region derived from the determined target region, and transmits it to the server device 130 as information on the region in association with the first encoded data. · Notify the determined limit quantization value to the first encoding unit 1020, and transmit it to the server device 130 as information regarding the quantization value, in association with the first encoded data. Also, notify the determined predetermined quantization value to the second encoding unit 1030, and transmit it to the server device 130 as information regarding the quantization value, in association with the second encoded data.

[0126] Also, when the recognition target is not included in the image data, the compression information determination unit 1010 · Notify the entire area to the area separation unit 320, and transmit it to the server device 130 as information regarding the area, in association with the first encoded data and the second encoded data. · Notify the determined predetermined quantization value to the first encoding unit 1020 and the second encoding unit 1030, and transmit it to the server device 130 as information regarding the quantization value, in association with the first encoded data and the second encoded data.

[0127] The first encoding unit 1020 encodes the first image data notified from the area separation unit 320 using the limit quantization value (or the predetermined quantization value) notified from the compression information determination unit 1010, and generates the first encoded data. Also, the first encoding unit 1020 transmits the generated first encoded data to the server device 130.

[0128] The second encoding unit 1030 encodes the second image data notified from the area separation unit 320 using the predetermined quantization value notified from the compression information determination unit 1010, and generates the second encoded data. Also, the second encoding unit 1030 transmits the generated second encoded data to the server device 130.

[0129] <Functional Configuration of the Transcoding Unit> Next, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the third embodiment will be described with reference to FIG. 10B. FIG. 10B is a third diagram showing an example of the functional configuration of the transcoding unit.

[0130] The difference from FIG. 4 is that the functions of the reconstruction unit 1040 and the quantization value map generation unit 1050 are different from those of the reconstruction unit 430 and the quantization value map generation unit 440 in FIG. 4.

[0131] The reconstruction unit 1040 extracts an image of the target region from the first decoded data based on the information about the region transmitted from the hierarchical encoding device 111. Also, the reconstruction unit 1040 extracts an image of the non-target region from the second decoded data based on the information about the region transmitted from the hierarchical encoding device 111. Further, the reconstruction unit 1040 synthesizes the extracted image of the target region and the extracted image of the non-target region to generate reconstructed image data. Furthermore, the reconstruction unit 1040 notifies the re-encoding unit 450 of the generated reconstructed image data.

[0132] The quantization value map generation unit 1050 generates a quantization value map based on the information about the region and the quantization value transmitted from the hierarchical encoding device 111. In the quantization value map generation unit 1050, a quantization value map is generated by setting a limit quantization value or a quantization value close to the limit quantization value for the target region and setting a predetermined quantization value for the non-target region.

[0133] Also, the quantization value map generation unit 1050 notifies the re-encoding unit 450 of the generated quantization value map.

[0134] <Flow of image processing by the image processing system> Next, the flow of image processing by the image processing system 100 according to the third embodiment will be described. FIG. 11 is a third flowchart showing the flow of image processing. The differences from FIG. 6 are steps S1101, S1102, S1103 to S1105.

[0135] In step S1101, the hierarchical encoding device 111 determines the target region and the non-target region for the image data of each frame included in the moving image, and transmits the determined target region and non-target region to the server device 130 as information about the region.

[0136] In step S1102, the hierarchical encoding device 111 determines the limit quantization value of the target region and a predetermined quantization value of the non-target region for the image data of each frame included in the moving image data. Further, the hierarchical encoding device 111 transmits the determined limit quantization value and the predetermined quantization value to the server device 130 as information regarding the quantization value.

[0137] In step S1103, the transcoding unit 121 of the server device 130 extracts the image of the target region from the first decoded data and the image of the non-target region from the second decoded data based on the information regarding the region transmitted from the hierarchical encoding device 111. Further, the transcoding unit 121 of the server device 130 synthesizes the extracted image of the target region and the extracted image of the non-target region to generate reconstructed image data.

[0138] In step S1104, the transcoding unit 121 of the server device 130 acquires the information regarding the region and the quantization value transmitted from the hierarchical encoding device 111.

[0139] In step S1105, the transcoding unit 121 of the server device 130 generates a quantization value map having different quantization values for the target region and the non-target region based on the acquired information regarding the region and the quantization value.

[0140] As is clear from the above description, in the image processing system 100 according to the third embodiment, the hierarchical encoding device 111 transmits the information regarding the region and the quantization value to the server device 130 separately from the first encoded data and the second encoded data.

[0141] Thereby, according to the image processing system 100 according to the third embodiment, the same effects as those of the first embodiment can be enjoyed.

[0142] [Fourth Embodiment] In each of the above embodiments, the case where the quantization value map generation unit acquires information regarding the region and the quantization value and generates a quantization value map based on the acquired information regarding the region and the quantization value has been described. However, the method for generating the quantization value map is not limited to this.

[0143] For example, assume that instead of the compression information determination unit 310 of the hierarchical encoding device 111 setting quantization values for the first encoding unit 330 and the second encoding unit 340, the bit rates are set for the first encoding unit 330 and the second encoding unit 340 to control the quantization values.

[0144] In this case, the reconstruction unit 430 cannot acquire information regarding the quantization values. Therefore, in the fourth embodiment, first, the bit rates of the first encoded data and the second encoded data transmitted from the hierarchical encoding device 111 are acquired, and the re-bit rate of the re-encoded data transmitted by the re-encoding unit 450 is determined. Subsequently, in the fourth embodiment, a quantization value map is generated such that the bit rate of the re-encoded data becomes the determined re-bit rate. Hereinafter, the fourth embodiment will be described centering on the differences from the above embodiments.

[0145] <Functional configuration of the transcoding unit> First, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the fourth embodiment will be described with reference to FIG. 12. FIG. 12 is a fourth diagram showing an example of the functional configuration of the transcoding unit.

[0146] The differences from FIG. 4 are that it has a bit rate acquisition unit 1210 and the function of the quantization value map generation unit 1220 is different from the function of the quantization value map generation unit 440 shown in FIG. 4.

[0147] The bit rate acquisition unit 1210 acquires the bit rate (first bit rate) of the first encoded data transmitted by the hierarchical encoding device 111 from the first decoding unit 410. Further, the bit rate acquisition unit 1210 acquires the bit rate (second bit rate) of the second encoded data transmitted by the hierarchical encoding device 111 from the second decoding unit 420.

[0148] Also, based on the acquired first bit rate and second bit rate, the bit rate acquisition unit 1210 determines the re-bit rate of the re-encoded data transmitted by the re-encoding unit 450. Further, the bit rate acquisition unit 1210 notifies the determined re-bit rate to the quantization value map generation unit 1220.

[0149] The quantization value map generation unit 1220 generates a quantization value map based on the re-bit rate determined by the bit rate acquisition unit 1210 and the information regarding the region notified by the reconstruction unit 430. Also, the quantization value map generation unit 1220 notifies the generated quantization value map to the re-encoding unit 450.

[0150] <Flow of image processing by the image processing system> Next, the flow of image processing by the image processing system 100 according to the fourth embodiment will be described. FIG. 13 is a fourth flowchart showing the flow of image processing. The differences from FIG. 6 are steps S1301 and S1302.

[0151] In step S1301, the transcoding unit 121 of the server device 130 acquires the bit rates (first and second bit rates) of the first encoded data and the second encoded data transmitted from the hierarchical encoding device 111.

[0152] In step S1302, the transcoding unit 121 of the server device 130 determines the re-bit rate of the re-encoded data based on the acquired bit rates (the first and second bit rates). Further, the transcoding unit 121 of the server device 130 generates a quantization value map based on the determined re-bit rate and the information regarding the area.

[0153] As is clear from the above description, in the image processing system 100 according to the fourth embodiment, the quantization value map generation unit generates a quantization value map from the re-bit rate determined based on the first bit rate and the second bit rate.

[0154] Thereby, according to the image processing system 100 according to the fourth embodiment, even when the information regarding the quantization value cannot be acquired from the hierarchical encoding device 111, the re-encoded data can be generated with the same quantization value as that of the hierarchical encoding device 111. That is, according to the image processing system 100 according to the fourth embodiment, the bit rate can be maintained before and after the transcoding unit 121.

[0155] As a result, according to the image processing system 100 according to the fourth embodiment, the same effect as that of the first embodiment can be enjoyed, and even when the information regarding the quantization value cannot be acquired, the occurrence of transmission delay can be avoided.

[0156] In the above description, it has been described that the first bit rate of the first encoded data and the second bit rate of the second encoded data are actually measured by the transcoding unit 121.

[0157] However, the first bit rate and the second bit rate acquired by the bit rate acquisition unit 1210 are not limited to the actually measured bit rates. For example, the first bit rate and the second bit rate acquired by the bit rate acquisition unit 1210 may be the bit rates set by the compression information determination unit 310 for the first encoding unit 330 and the second encoding unit 340.

[0158] Further, the first bit rate and the second bit rate acquired by the bit rate acquisition unit 1210 are not limited to the bit rates actually measured by the transcoding unit 121. For example, the bit rate acquisition unit 1210 may acquire the first bit rate and the second bit rate actually measured by the hierarchical encoding device 111.

[0159] [Fifth Embodiment] In the above fourth embodiment, the case of generating the quantization value map based on the bit rates (first and second bit rates) of the first encoded data and the second encoded data transmitted from the hierarchical encoding device 111 has been described. However, the method of generating the quantization value map is not limited to this. For example, a quantization value map may be generated based on information regarding the region and the quantization value, and further, the quantization value map may be corrected based on the ratio between the bit rates (first and second bit rates) of the first encoded data and the second encoded data and the re-bit rate of the re-encoded data. Hereinafter, the fifth embodiment will be described centering on the differences from the above-described embodiments.

[0160] [Functional Configuration of Transcoding Unit] First, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the fifth embodiment will be described with reference to FIG. 14. FIG. 14 is a fifth diagram showing an example of the functional configuration of the transcoding unit.

[0161] The differences from FIG. 4 are that it has a correction coefficient calculation unit 1410 and that the function of the quantization value map generation unit 1420 is different from the function of the quantization value map generation unit 440 shown in FIG. 4.

[0162] The correction coefficient calculation unit 1410 acquires the first bit rate, which is the bit rate of the first encoded data, from the first decoding unit 410. Further, the correction coefficient calculation unit 1410 acquires the second bit rate, which is the bit rate of the second encoded data, from the second decoding unit 420.

[0163] Further, the correction coefficient calculation unit 1410 obtains a re-bit rate, which is the bit rate when the re-encoding unit 450 transmits the re-encoded data to the re-encoded data acquisition unit 131.

[0164] Furthermore, the correction coefficient calculation unit 1410 calculates a correction coefficient α for correcting the quantization value map based on the acquired first bit rate, second bit rate, and re-bit rate, and notifies the quantization value map generation unit 1420.

[0165] The quantization value map generation unit 1420 generates a quantization value map based on the information on the region and quantization value notified from the reconstruction unit 430. At this time, the quantization value map generation unit 1420 generates a quantization value map by setting a limit quantization value or a quantization value close to the limit quantization value in the target region and setting a predetermined quantization value in the non-target region.

[0166] In addition, the quantization value map generation unit 1420 corrects the target region of the generated quantization value map using the correction coefficient α notified from the correction coefficient calculation unit 1410. Furthermore, the quantization value map generation unit 1420 notifies the re-encoding unit 450 of the corrected quantization value map.

[0167] <Specific Example of the Processing of the Correction Coefficient Calculation Unit> Next, a specific example of the processing of the correction coefficient calculation unit 1410 will be described. FIG. 15 is a diagram showing a specific example of the processing of the correction coefficient calculation unit. As shown in the example of FIG. 15, the correction coefficient calculation unit 1410 calculates the correction coefficient α based on the following formula (1).

[0168]

Equation

[0169] As shown in FIG. 15, the correction coefficient α calculated by the correction coefficient calculation unit 1410 is applied to the target region of the quantization value map 1510 generated by the quantization value map generation unit 1420. As a result, the quantization value map 1510 is corrected, and a corrected quantization value map 1520 is generated.

[0170] <Flow of Image Processing by Image Processing System> Next, the flow of image processing by the image processing system 100 according to the fifth embodiment will be described. FIG. 16 is a fifth flowchart showing the flow of image processing. The differences from FIG. 13 are steps S1601 to S1603.

[0171] In step S1601, the transcoding unit 121 of the server device 130 acquires the re-bit rate of the re-encoded data transmitted by the re-encoding unit 450.

[0172] In step S1602, the transcoding unit 121 of the server device 130 calculates a correction coefficient α based on the bit rate (first and second bit rates) acquired in step S1301 and the re-bit rate acquired in step S1601.

[0173] In step S1603, the transcoding unit 121 of the server device 130 corrects the quantization value map by multiplying the target region of the quantization value map generated in step S610 by the correction coefficient α calculated in step S1602. As a result, the transcoding unit 121 of the server device 130 generates a corrected quantization value map.

[0174] As is clear from the above description, in the image processing system 100 according to the fifth embodiment, the quantization value map generation unit corrects the quantization value map generated based on the information regarding the region and the quantization value based on the ratio of the first and second bit rates and the re-bit rate.

[0175] Accordingly, according to the image processing system 100 according to the fifth embodiment, the quantization value map can be corrected according to the ratio between the first and second bitrates and the rebite rate.

[0176] As a result, according to the image processing system 100 according to the fifth embodiment, the same effects as those of the first embodiment can be enjoyed, and the occurrence of transmission delay can be avoided.

[0177] [Sixth Embodiment] In the above-described first to third and fifth embodiments, the case of generating a quantization value map based on information regarding the region and quantization values has been described, and in the above-described fourth embodiment, the case of generating a quantization value map based on the bitrate has been described. However, the method for generating the quantization value map is not limited to these. For example, a quantization value map may be generated based on the attributes of the image data of each frame included in the moving image data captured by the imaging device 110 and the attributes of the corresponding reconstructed image data generated by the reconstruction unit 430. Hereinafter, the sixth embodiment will be described centering on the differences from the above-described embodiments.

[0178] <Functional Configuration of Hierarchical Encoding Device> First, the functional configuration of the hierarchical encoding device 111 of the image processing system 100 according to the sixth embodiment will be described with reference to FIG. 17A. FIG. 17A is a third diagram showing an example of the functional configuration of the hierarchical encoding device. The difference from FIG. 3 is that it includes an image MAD calculation unit 1710.

[0179] The image MAD calculation unit 1710 calculates an image MAD (Mean absolute deviation) value for each encoding block of the image data of each frame included in the moving image data. Further, the calculated image MAD values of the respective encoding blocks are transmitted to the server device 130. Note that the MAD value refers to the dispersion of the pixel values in the image data, and the image MAD calculation unit 1710 calculates the image MAD value of the encoding block based on, for example, the following formula (2).

[0180]

Number

[0181] <Functional Configuration of the Transcoding Unit> Next, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the sixth embodiment will be described with reference to FIG. 17B. FIG. 17B is a sixth diagram showing an example of the functional configuration of the transcoding unit.

[0182] The differences from FIG. 4 are that it has a reconstructed image MAD calculation unit 1720 and a quantization value calculation unit 1730, and the function of the quantization value map generation unit 1740 is different from the function of the quantization value map generation unit 440 shown in FIG. 4.

[0183] The reconstructed image MAD calculation unit 1720 calculates the reconstructed image MAD value for each encoding block based on the reconstructed image data generated by the reconstruction unit 430. Further, it notifies the quantization value calculation unit 1730 of the calculated reconstructed image MAD value for each encoding block. In the reconstructed image MAD calculation unit 1720, for example, the reconstructed image MAD value of the encoding block is calculated based on the following formula (3).

[0184]

Number

[0185] The quantization value calculation unit 1730 calculates the quantization value for each encoding block based on the image MAD value of each encoding block transmitted from the hierarchical encoding device 111 and the reconstructed image MAD value of each encoding block notified from the reconstructed image MAD calculation unit 1720.

[0186] Further, the quantization value calculation unit 1730 notifies the quantization value map generation unit 1740 of the calculated quantization values of the respective encoded blocks.

[0187] The quantization value map generation unit 1740 generates a quantization value map based on the quantization values of the respective encoded blocks transmitted from the quantization value calculation unit 1730, and notifies the re-encoding unit 450.

[0188] <Specific Example of the Processing of the Quantization Value Calculation Unit> Next, a specific example of the processing of the quantization value calculation unit 1730 will be described. FIG. 18 is a first diagram showing a specific example of the processing of the quantization value calculation unit. As shown in FIG. 18, the quantization value calculation unit 1730 further includes an MAD difference calculation unit 1810, and calculates the difference between the image MAD value transmitted from the hierarchical encoding device 111 and the reconstructed image MAD value notified from the reconstructed image MAD calculation unit 1720.

[0189] Here, the difference between the image MAD value and the reconstructed image MAD value has a conversion relationship with the PSNR (Peak Signal to Noise Ratio). Therefore, the MAD difference calculation unit 1810 can calculate the PSNR based on the difference between the image MAD value and the reconstructed image MAD value.

[0190] Also, as shown in FIG. 18, the quantization value calculation unit 1730 further includes a quantization value conversion unit 1820, and calculates a quantization value based on the PSNR calculated by the MAD difference calculation unit 1810.

[0191] Here, there is a relationship between the PSNR and the quantization value as shown in the graph 1821. Therefore, the quantization value conversion unit 1820 can calculate and output the quantization value from the PSNR by referring to the graph 1821.

[0192] Note that the quantization value calculation unit 1730 performs the above-described processing for each encoded block to output the quantization value for each encoded block.

[0193] <Flow of Image Processing by the Image Processing System> Next, the flow of image processing by the image processing system 100 according to the sixth embodiment will be described. FIG. 19 is a sixth flowchart showing the flow of image processing. The differences from FIG. 6 are steps S1901, S1902 to S1904.

[0194] In step S1901, the hierarchical encoding device 111 calculates the image MAD value for each encoding block of the image data of each frame included in the moving image data, and transmits it to the server device 130.

[0195] In step S1902, the transcoding unit 121 of the server device 130 calculates the reconstructed image MAD value for each encoding block of the reconstructed image data.

[0196] In step S1903, the transcoding unit 121 of the server device 130 calculates the difference between the image MAD value and the reconstructed image MAD value for each encoding block, and calculates a quantization value from the PSNR value corresponding to the difference.

[0197] In step S1904, the transcoding unit 121 of the server device 130 generates a quantization value map using the quantization value for each encoding block.

[0198] As is clear from the above description, the image processing system 100 according to the sixth embodiment calculates a quantization value for each encoding block based on the difference between the image MAD value and the reconstructed image MAD value, and generates a quantization value map.

[0199] Thereby, according to the image processing system 100 according to the sixth embodiment, a quantization value map can be generated based on the attributes of the image data captured by the imaging device 110 and the attributes of the reconstructed image data generated by the reconstruction unit 430.

[0200] As a result, according to the image processing system 100 according to the sixth embodiment, the same effects as those of the first embodiment can be enjoyed.

[0201] [Embodiment 7] In the above-described sixth embodiment, the case of generating a quantization value map by calculating the PSNR based on the attributes of the image data and the attributes of the reconstructed image data and calculating the quantization value from the calculated PSNR was described. However, the generation method for generating the quantization value map based on the attributes of the image data and the attributes of the reconstructed image data is not limited to the generation method described in the sixth embodiment. Further, in the above-described sixth embodiment, the case of applying the generated quantization value map to all the reconstructed image data was described. However, the application destination of the generated quantization value map is not limited to the application destination (all the reconstructed image data) described in the sixth embodiment. Hereinafter, the seventh embodiment will be described focusing on the differences from the sixth embodiment.

[0202] <Specific Examples of the Processing of the Quantization Value Calculation Unit and the Processing of the Quantization Value Map Generation Unit> First, specific examples of the processing of the quantization value calculation unit 1730 and the processing of the quantization value map generation unit 1740 in the seventh embodiment will be described. FIG. 20 is a diagram showing specific examples of the processing of the quantization value calculation unit and the quantization value map generation unit. As shown in FIG. 20, the quantization value calculation unit 1730 further includes a MAD difference calculation unit 1810 and a quantization value conversion unit 2010. Among these, since the MAD difference calculation unit 1810 is the same as the MAD difference calculation unit 1810 described with reference to FIG. 18 in the sixth embodiment, the description thereof will be omitted here.

[0203] The quantization value conversion unit 2010 directly calculates the quantization value for each coding block based on the difference between the image MAD value and the reconstructed image MAD value calculated by the MAD difference calculation unit 1810. In the above-described sixth embodiment, the PSNR was calculated based on the difference between the image MAD value and the reconstructed image MAD value, and the quantization value was calculated from the calculated PSNR.

[0204] In contrast, in the seventh embodiment, the relationship between the difference between the image MAD value and the reconstructed image MAD value and the quantization value is obtained in advance (see reference numeral 2011), and based on this relationship, the quantization value for each encoded block is directly calculated from the difference between the image MAD value and the reconstructed image MAD value.

[0205] Also, the quantization value conversion unit 2010 notifies the quantization value map generation unit 1740 of the calculated quantization value for each encoded block.

[0206] As shown in FIG. 20, the quantization value map generation unit 1740 further includes a quantization value adjustment unit 2020 and a mapping unit 2030, and the quantization value for each encoded block notified by the quantization value conversion unit 2010 is input to the quantization value adjustment unit 2020 and the mapping unit 2030.

[0207] Here, in the quantization value map generation unit 1740, when the corresponding image data is an I picture, a quantization value map is generated using the quantization value for each encoded block notified by the quantization value conversion unit 2010.

[0208] Also, in the quantization value map generation unit 1740, when the corresponding image data is a P picture, basically, a quantization value map is generated using the quantization value applied to the previous P picture. However, when the number of encoded blocks to which the intra prediction mode is applied during encoding is large, the quantization value map generation unit 1740 · the quantization value for each encoded block notified by the quantization value conversion unit 2010, and · the quantization value applied to the previous P picture, are used to generate a quantization value map.

[0209] This will be specifically described with reference to FIG. 20. When the corresponding image data is an I picture, the mapping unit 2030 generates a quantization value map using the quantization values for each encoded block notified by the quantization value conversion unit 2010. Also, when the image data is a P picture, the mapping unit 2030 generates a quantization value map using the quantization values for each encoded block notified by the quantization value adjustment unit 2020. Further, the mapping unit 2030 notifies the generated quantization value map to the re-encoding unit 450 and stores it in the quantization value storage unit 2040.

[0210] When the image data is a P picture, the quantization value adjustment unit 2020 refers to the quantization value storage unit 2040 and reads out the quantization values applied to the previous P picture from the quantization value storage unit 2040. Also, the quantization value adjustment unit 2020 notifies the read quantization values to the mapping unit 2030.

[0211] However, when the image data is a P picture and the number of encoded blocks to which the intra prediction mode is applied during encoding is large, the quantization value adjustment unit 2020 · uses the quantization values for each encoded block notified by the quantization value conversion unit 2010, and · the quantization values applied to the previous P picture, and adjusts the quantization values and notifies the adjusted quantization values to the mapping unit 2030.

[0212] In FIG. 20, reference numeral 2021 is a graph showing the relationship between the image data and the first bit rate (the bit rate of the first encoded data). Among the image data of each frame included in the moving image data, in the graph shown by reference numeral 2021, the image data with a high first bit rate is the image data in which the number of encoded blocks to which the intra prediction mode is applied is large. Note that the encoded blocks to which the intra prediction mode is applied are the encoded blocks in the moving region, the encoded blocks in the boundary region between the target region and the non-target region, and the like.

[0213] In the case of the example of symbol 2011, in the quantization value adjustment unit 2020, the quantization value is adjusted for the image data of the P picture indicated by symbol 2022.

[0214] <Flow of image processing by the image processing system> Next, the flow of image processing by the image processing system 100 according to the seventh embodiment will be described. FIG. 21 is a seventh flowchart showing the flow of image processing. The differences from FIG. 19 are steps S2101 and S2102.

[0215] In step S2101, when the image data is an I picture, the transcoding unit 121 of the server device 130 generates a quantization value map using the quantization value for each encoding block calculated in step S1903.

[0216] In step S2102, when the image data is a P picture, the transcoding unit 121 of the server device 130 generates a quantization value map using the quantization value applied to the previous P picture. However, when the number of encoding blocks to which the intra prediction mode was applied during encoding is large, the transcoding unit 121 of the server device 130 adjusts the quantization value applied to the previous P picture using the quantization value calculated this time (step S1903). Then, the transcoding unit 121 of the server device 130 generates a quantization value map using the adjusted quantization value.

[0217] As is clear from the above description, when the image processing system 100 according to the seventh embodiment generates a quantization value map based on the attributes of the image data and the attributes of the reconstructed image data, · Calculate the quantization value directly from the difference between the two. · Generate different quantization value maps for I pictures and P pictures. · For P pictures, generate different quantization value maps according to the prediction mode during encoding.

[0218] Accordingly, according to the image processing system 100 according to the seventh embodiment, a quantization value map suitable for the content of the encoding process can be generated.

[0219] As a result, according to the image processing system 100 according to the seventh embodiment, the same effects as those of the first embodiment can be enjoyed, and an appropriate quantization value map can be generated.

[0220] [Eighth Embodiment] In the sixth and seventh embodiments described above, the case where the quantization value is determined based on the attributes of the image data and the attributes of the reconstructed image data, and the quantization value map is generated has been described. However, the method for generating the quantization value map is not limited to this. For example, the quantization value may be determined based on the attributes of the reconstructed image data so that the bit rate (re-bit rate) of the reconstructed image data approaches the target bit rate, and the quantization value map may be generated. Hereinafter, the eighth embodiment will be described centering on the differences from the sixth and seventh embodiments.

[0221] [Functional Configuration of Transcoding Unit] First, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the eighth embodiment will be described with reference to FIG. 22. FIG. 22 is a seventh diagram showing an example of the functional configuration of the transcoding unit.

[0222] The difference from FIG. 17B is that the functions of the quantization value calculation unit 2210 and the quantization value map generation unit 2220 are different from the functions of the quantization value calculation unit 1730 and the quantization value map generation unit 1740 shown in FIG. 17B.

[0223] The quantization value calculation unit 2210 determines a quantization value based on the reconstructed image MAD value notified by the reconstructed image MAD calculation unit 1720. Further, the quantization value calculation unit 2210 notifies the determined quantization value to the quantization value map generation unit 2220.

[0224] Note that the quantization value calculation unit 2210 may determine the quantization values of all the encoding blocks, or may determine the quantization values of the encoding blocks corresponding to the target region. FIG. 22 shows the case where the quantization value calculation unit 2210 determines the quantization values of the encoding blocks corresponding to the target region. Specifically, the quantization value calculation unit 2210 determines the quantization values of the encoding blocks corresponding to the target region such that the re-bit rate of the re-encoded data predicted based on the reconstructed image MAD value becomes the target bit rate.

[0225] In addition, the quantization value calculation unit 2210 notifies the quantization value map generation unit 2220 of the determined quantization values.

[0226] The quantization value map generation unit 2220 generates a quantization value map based on the information on the region and the quantization values notified from the reconstruction unit 430. In addition, the quantization value map generation unit 2220 corrects the quantization values of the encoding blocks corresponding to the target region in the generated quantization value map with the quantization values notified from the quantization value calculation unit 2210, and notifies the re-encoding unit 450 of the corrected quantization value map.

[0227] <Specific Example of the Processing of the Quantization Value Calculation Unit> Next, a specific example of the processing of the quantization value calculation unit 2210 will be described. FIG. 23 is a second diagram showing a specific example of the processing of the quantization value calculation unit. As shown in FIG. 23, the quantization value calculation unit 2210 includes a prediction unit 2310 and acquires the reconstructed image MAD value from the reconstructed image MAD calculation unit 1720.

[0228] The prediction unit 2310 pre-holds the relationship between the reconstructed image MAD value and the re-bit rate for each quantization value, and based on the relationship, predicts the re-bit rate of the re-encoded data when each quantization value is used from the acquired reconstructed image MAD value. In addition, the prediction unit 2310 determines the quantization value for which the predicted re-bit rate becomes the target bit rate, and notifies the quantization value map generation unit 2220 of the determined quantization value as the quantization value of the encoding block corresponding to the target region.

[0229] <Flow of Image Processing by Image Processing System> Next, the flow of image processing by the image processing system 100 according to the eighth embodiment will be described. FIG. 24 is an eighth flowchart showing the flow of image processing. The differences from FIG. 6 are steps S2401 to S2403.

[0230] In step S2401, the transcoding unit 121 of the server device 130 calculates the reconstructed image MAD value for each coding block corresponding to the target region among the reconstructed image data.

[0231] In step S2402, the transcoding unit 121 of the server device 130 predicts the recoding data's re-bit rate when encoding using each quantization value based on the calculated reconstructed image MAD value.

[0232] In step S2403, the transcoding unit 121 of the server device 130 determines the quantization value corresponding to the re-bit rate closest to the target bit rate among the predicted re-bit rates. Also, the transcoding unit 121 of the server device 130 corrects the quantization value of the coding block corresponding to the target region in the quantization value map generated in step S610 using the determined quantization value, and generates a corrected quantization value map.

[0233] As is clear from the above description, the image processing system 100 according to the eighth embodiment corrects the quantization value map so that the re-bit rate of the recoding data predicted based on the reconstructed image MAD value approaches the target bit rate.

[0234] Thereby, according to the image processing system 100 according to the eighth embodiment, the re-bit rate of the recoding data can be controlled to the target bit rate.

[0235] As a result, according to the image processing system 100 according to the eighth embodiment, the same effects as those of the first embodiment can be enjoyed, and the occurrence of transmission delay can be avoided.

[0236] [Embodiment 9] In the above-described first embodiment, the case where the quantization value map generation unit generates a quantization value map based on information regarding the region and the quantization value has been described. However, the method for generating the quantization value map is not limited to this. For example, a quantization value map may be generated using the minimum value of the information regarding the region and the quantization value calculated for the image data of each frame included in the moving image data. Hereinafter, the ninth embodiment will be described centering on the differences from the first embodiment.

[0237] [Functional Configuration of Transcoding Unit] FIG. 25 is a eighth diagram showing an example of the functional configuration of the transcoding unit, and is an example of the functional configuration in the case of generating a quantization value map using the minimum value of the information regarding the region and the quantization value.

[0238] The difference from FIG. 4 is that it has a minimum value calculation unit 2510. The minimum value calculation unit 2510 · Calculates the minimum quantization value of the target region by calculating the minimum value of the information regarding the region and the quantization value (target region and limit quantization value) notified from the first decoding unit 410 for the image data of a predetermined number of frames, · Calculates the minimum quantization value of the non-target region by calculating the minimum value of the information regarding the region and the quantization value (non-target region and predetermined quantization value) notified from the second decoding unit 420 for the image data of a predetermined number of frames.

[0239] Further, the minimum value calculation unit 2510 notifies the calculated minimum quantization value to the quantization value map generation unit 440.

[0240] [Flow of Image Processing by Image Processing System] Next, the flow of image processing by the image processing system 100 according to the ninth embodiment will be described. FIG. 26 is a ninth flowchart showing the flow of image processing. The differences from FIG. 6 are steps S2601 and S2602.

[0241] In step S2601, the transcoding unit 121 of the server device 130 calculates the minimum quantization values of the target region and the non-target region by calculating the minimum value of the information regarding the region and the quantization value.

[0242] In step S2602, the transcoding unit 121 of the server device 130 generates a quantization value map using the minimum quantization values.

[0243] In this way, by effectively using the information regarding the region and the quantization value determined when the hierarchical encoding device 111 performs the encoding process also when the re-encoding unit 450 generates the re-encoded data, appropriate re-encoded data can be generated.

[0244] Note that the method of effectively using the information regarding the region and the quantization value determined when the hierarchical encoding device 111 performs the encoding process is not limited to the above description. For example, when the transcoding unit 121 can directly obtain the quantization value maps used when the first encoding unit 330 and the second encoding unit 340 encode the image data respectively, the obtained quantization value maps may be used to generate the re-encoded data.

[0245] Also, when the encoding methods of the first encoding unit 330 and the second encoding unit 340 are different from the encoding method of the re-encoding unit 450, after performing a predetermined correction on the obtained quantization value maps, the re-encoded data may be generated.

[0246] Note that in the above description, the minimum value of the information regarding the region and the quantization value calculated for the image data of each frame included in the moving image data is used, but an average value may also be used. Also, among the information regarding the region and the quantization value in the target region, the information corresponding to the outlier may be excluded when calculating the minimum quantization value or the average quantization value. Alternatively, among the information regarding the region and the quantization value in the non-target region, the information corresponding to the outlier may be excluded when calculating the minimum quantization value or the average quantization value.

[0247] [Embodiment 10] In the fifth embodiment described above, the case where the correction coefficient α is calculated based on the bit rates of the first encoded data and the second encoded data and the bit rate of the re-encoded data has been described. However, the method for calculating the correction coefficient α is not limited to this. For example, the correction coefficient α may be calculated using the PSNR calculated for the re-decoded data. Hereinafter, the tenth embodiment will be described centering on the differences from the above-described embodiments.

[0248] <Functional Configuration of Transcoding Unit>[[]] First, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the tenth embodiment will be described with reference to FIG. 27. FIG. 27 is the ninth diagram showing an example of the functional configuration of the transcoding unit. The difference from FIG. 14 is that it includes a re-decoding unit 2710 and a PSNR calculation unit 2720, and the function of the correction coefficient calculation unit 2730 is different from the function of the correction coefficient calculation unit 1410 shown in FIG. 14.

[0249] The re-decoding unit 2710 re-decodes the re-encoded data generated by the re-encoding unit 450 and generates re-decoded data. The re-decoding unit 2710 notifies the generated re-decoded data to the PSNR calculation unit 2720.

[0250] The PSNR calculation unit 2720 calculates the PSNR of the re-decoded data notified from the re-decoding unit 2710 and notifies the calculated PSNR to the correction coefficient calculation unit 2730.

[0251] The correction coefficient calculation unit 2730 calculates the correction coefficient α based on the PSNR calculated for the re-decoded data corresponding to the previous image data and the PSNR calculated for the re-decoded data corresponding to the current image data. Further, the correction coefficient calculation unit 2730 notifies the calculated correction coefficient α to the quantization value map generation unit 1420.

[0252] <Relationship between Quantization Value of Quantization Value Map and PSNR>[[]] Here, the relationship between the quantization value of the quantization value map and the PSNR will be briefly described. The reconstructed image data re-encoded by the re-encoding unit 450 using the quantization value map is generated based on the first decoded data and the second decoded data, and some information is lost when the first encoding unit 330 and the second encoding unit 340 perform encoding.

[0253] On the other hand, when the re-encoding unit 450 re-encodes the reconstructed image data, even if the quantization value of the quantization value map is reduced, some of the already lost information will not be restored.

[0254] Therefore, when re-encoding the reconstructed image data, as the quantization value of the quantization value map is reduced, there exists a quantization value at which the image quality of the re-decoded data will no longer improve (that is, a quantization value at which the PSNR will no longer improve).

[0255] Also, even if the quantization value of the quantization value map is reduced more than necessary, the data volume of the re-encoded data will not increase extremely. In addition, if the quantization value of the quantization value map is reduced more than necessary, the encoding noise added when the first encoding unit 330 and the second encoding unit 340 perform encoding may be reproduced, and instead, the image quality may deteriorate.

[0256] From the above, when correcting the quantization value map, it is desirable to correct the quantization value map so that the quantization value is not smaller than the quantization value at which the PSNR will no longer improve.

[0257] <Specific Example of the Processing of the Correction Coefficient Calculation Unit> Subsequently, a specific example of the processing of the correction coefficient calculation unit 2730 will be described based on the relationship between the quantization value of the quantization value map and the PSNR. FIG. 28 is a second diagram showing a specific example of the processing of the correction coefficient calculation unit. As shown in the example of FIG. 28, the correction coefficient calculation unit 2730 calculates the correction coefficient α based on the following formula (4).

[0258]

Equation

[0259] On the other hand, when the PSNR of the decoded data corresponding to the previous image data is better than the PSNR of the decoded data corresponding to the current image data, a correction coefficient α greater than or equal to 1 is calculated. Therefore, the quantization value of the corrected quantization value map becomes larger than before the correction.

[0260] Note that in the above formula (4), the reactivity is a parameter for gradually reflecting, without directly reflecting, the ratio between the PSNR of the decoded data corresponding to the previous image data and the PSNR of the decoded data corresponding to the current image data in the quantization value.

[0261] As shown in FIG. 28, the correction coefficient α calculated by the correction coefficient calculation unit 2730 is multiplied by the quantization value map 1510 generated by the quantization value map generation unit 1420. By correcting the quantization value map 1510, a corrected quantization value map 1520 is generated.

[0262] <Flow of Image Processing by Image Processing System> Next, the flow of image processing by the image processing system 100 according to the tenth embodiment will be described. FIG. 29 is a tenth flowchart showing the flow of image processing. The differences from FIG. 16 are steps S2901 and S2902.

[0263] In step S2901, the transcoding unit 121 of the server device 130 decodes the re-encoded data and calculates the PSNR.

[0264] In step S2902, the transcoding unit 121 of the server device 130 calculates a correction coefficient α using the PSNR calculated for the decoded data corresponding to the previous image data and the PSNR calculated for the decoded data corresponding to the current image data.

[0265] As is clear from the above description, in the image processing system 100 according to the tenth embodiment, the quantization value map generated by the quantization value map generation unit based on the information regarding the region and the quantization value is corrected based on the PSNR of the decoded data corresponding to the previous and current image data.

[0266] Thereby, according to the image processing system 100 according to the tenth embodiment, the quantization value map can be appropriately corrected based on the change in the PSNR with respect to the change in the quantization value.

[0267] As a result, according to the image processing system 100 according to the tenth embodiment, the same effects as those of the first embodiment can be enjoyed, and an appropriate quantization value map can be generated.

[0268] [Eleventh Embodiment] In the above tenth embodiment, the case where the correction coefficient α is calculated using the PSNR calculated for the decoded data has been described, but the method for calculating the correction coefficient α is not limited to this. For example, the correction coefficient α may be calculated using the recognition rate calculated for the decoded data. Hereinafter, the eleventh embodiment will be described centering on the differences from the above tenth embodiment.

[0269] [Functional Configuration of Transcoding Unit] First, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the eleventh embodiment will be described with reference to FIG. 30. FIG. 30 is a tenth diagram showing an example of the functional configuration of the transcoding unit. The differences from FIG. 27 are that it includes a recognition unit 3010 instead of the PSNR calculation unit 2720, and the function of the correction coefficient calculation unit 3020 is different from the function of the correction coefficient calculation unit 2730 shown in FIG. 27.

[0270] The recognition unit 3010 calculates the recognition rate by performing recognition processing on the re-decoded data notified by the re-decoding unit 2710, and notifies the calculated recognition rate to the correction coefficient calculation unit 3020.

[0271] The correction coefficient calculation unit 3020 calculates a correction coefficient α based on the recognition rate calculated for the re-decoded data corresponding to the previous image data and the recognition rate calculated for the re-decoded data corresponding to the current image data. Further, the correction coefficient calculation unit 3020 notifies the calculated correction coefficient α to the quantization value map generation unit 1420.

[0272] As a result, the quantization value map can be corrected so that the quantization value map is not generated by a quantization value smaller than the quantization value at which the recognition rate does not improve any further.

[0273] <Specific Example of the Processing of the Correction Coefficient Calculation Unit> Next, a specific example of the processing of the correction coefficient calculation unit 3020 will be described. FIG. 31 is a third diagram showing a specific example of the processing of the correction coefficient calculation unit. As shown in FIG. 31, the correction coefficient calculation unit 3020 calculates the correction coefficient α based on the following formula (5).

[0274]

Equation

[0275] On the other hand, when the recognition rate of the re-decoded data corresponding to the previous image data is better than the recognition rate of the re-decoded data corresponding to the current image data, a correction coefficient α of 1 or more is calculated. Therefore, the quantization value of the corrected quantization value map becomes larger than before the correction.

[0276] Note that in the above formula (5), the reactivity is a parameter for ensuring that the ratio between the recognition rate of the decoded data corresponding to the previous image data and the recognition rate of the decoded data corresponding to the current image data is gradually reflected without being directly reflected in the quantization value.

[0277] As shown in FIG. 31, the correction coefficient α calculated by the correction coefficient calculation unit 3020 is multiplied by the quantization value map 1510 generated by the quantization value map generation unit 1420, and the quantization value map 1510 is corrected to generate a corrected quantization value map 1520.

[0278] <Flow of Image Processing by Image Processing System> Next, the flow of image processing by the image processing system 100 according to the 11th embodiment will be described. FIG. 32 is an 11th flowchart showing the flow of image processing. The differences from FIG. 29 are steps S3201 and S3202.

[0279] In step S3201, the transcoding unit 121 of the server device 130 decodes the re-encoded data and calculates the recognition rate by executing a recognition process.

[0280] In step S2902, the transcoding unit 121 of the server device 130 calculates the correction coefficient α using the recognition rate calculated for the decoded data corresponding to the previous image data and the recognition rate calculated for the decoded data corresponding to the current image data.

[0281] As is clear from the above description, in the image processing system 100 according to the 11th embodiment, the quantization value map generation unit corrects the quantization value map generated based on the information regarding the region and the quantization value based on the recognition rates of the decoded data corresponding to the previous and current image data.

[0282] Thereby, according to the image processing system 100 according to the 10th embodiment, the quantization value map can be appropriately corrected based on the change in the recognition rate with respect to the change in the quantization value.

[0283] As a result, according to the image processing system 100 according to the eleventh embodiment, the same effects as those of the first embodiment can be enjoyed, and an appropriate quantization value map can be generated.

[0284] [Twelfth Embodiment] In the tenth embodiment, the case where the quantization value map is appropriately corrected according to the change in PSNR has been described. However, the method for correcting the quantization value map using PSNR is not limited to this. For example, the quantization value map may be corrected so that the PSNR of the decoded data approaches the PSNR specified by the user. Hereinafter, the twelfth embodiment will be described centering on the differences from the tenth embodiment.

[0285] <Functional Configuration of the Transcoding Unit> First, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the twelfth embodiment will be described with reference to FIG. 33. FIG. 33 is the twelfth diagram showing an example of the functional configuration of the transcoding unit. The difference from FIG. 27 is that the function of the correction coefficient calculation unit 3310 is different from the function of the correction coefficient calculation unit 2730 shown in FIG. 27.

[0286] The correction coefficient calculation unit 3310 acquires in advance the PSNR specified by the user. Further, the correction coefficient calculation unit 3310 acquires the PSNR of the decoded data corresponding to the current image data calculated by the PSNR calculation unit 2720, and calculates the correction coefficient α by comparing it with the PSNR specified by the user. Further, the correction coefficient calculation unit 3310 notifies the quantization value map generation unit 1420 of the calculated correction coefficient α.

[0287] Thereby, the quantization value map can be corrected so as to approach the PSNR specified by the user.

[0288] <Specific Example of the Processing of the Correction Coefficient Calculation Unit> Next, a specific example of the processing of the correction coefficient calculation unit 3310 will be described. FIG. 34 is a fourth diagram showing a specific example of the processing of the correction coefficient calculation unit. As shown in the example of FIG. 34, the correction coefficient calculation unit 3310 calculates a correction coefficient α based on the following formula (6).

[0289] [Number] According to the above formula (6), when the PSNR of the decoded data corresponding to the current image data is greater than the PSNR specified by the user, a correction coefficient α less than 1 is calculated. Therefore, the quantization value of the corrected quantization value map becomes smaller than that before correction.

[0290] On the other hand, when the PSNR specified by the user is greater than the PSNR of the decoded data corresponding to the current image data, a correction coefficient α of 1 or more is calculated. Therefore, the quantization value of the corrected quantization value map becomes larger than that before correction.

[0291] In the above formula (6), the responsiveness is a parameter for gradually reflecting the ratio between the PSNR specified by the user and the PSNR of the decoded data corresponding to the current image data without directly reflecting it in the quantization value.

[0292] As shown in FIG. 34, the correction coefficient α calculated by the correction coefficient calculation unit 3310 is multiplied by the quantization value map 1510 generated by the quantization value map generation unit 1420, and the quantization value map 1510 is corrected to generate a corrected quantization value map 1520.

[0293] [Flow of Image Processing by Image Processing System] Next, the flow of image processing by the image processing system 100 according to the twelfth embodiment will be described. FIG. 35 is a twelfth flowchart showing the flow of image processing. The difference from FIG. 29 is step S3501.

[0294] In step S3501, the transcoding unit 121 of the server device 130 calculates a correction coefficient α based on the user-specified PSNR and the PSNR calculated for the decoded data corresponding to the current image data.

[0295] As is clear from the above description, in the image processing system 100 according to the twelfth embodiment, the quantization value map generated by the quantization value map generation unit based on the information regarding the region and the quantization value is corrected based on the user-specified PSNR and the PSNR of the decoded data.

[0296] Thereby, according to the image processing system 100 according to the twelfth embodiment, the quantization value map can be corrected so that the PSNR of the decoded data approaches the user-specified PSNR.

[0297] As a result, according to the image processing system 100 according to the twelfth embodiment, the same effects as those of the first embodiment can be obtained, and an appropriate quantization value map can be generated.

[0298] [Embodiment 13] In the above twelfth embodiment, the case where the quantization value map is corrected so that the PSNR of the decoded data approaches the user-specified PSNR has been described. However, the method for correcting the quantization value map is not limited to this, and the quantization value map may be corrected so that the re-bit rate of the re-encoded data generated by the re-encoding unit 450 approaches the user-specified bit rate. Hereinafter, the thirteenth embodiment will be described centering on the differences from the twelfth embodiment.

[0299] <Functional Configuration of Transcoding Unit> First, the functional configuration of the transcoding unit 121 of the server device 130 of the image processing system 100 according to the 13th embodiment will be described with reference to FIG. 36. FIG. 36 is the 13th diagram showing an example of the functional configuration of the transcoding unit. The differences from FIG. 33 are that it does not have a re-decoding unit 2710 and a PSNR calculation unit 2720, and the function of the correction coefficient calculation unit 3610 is different from the function of the correction coefficient calculation unit 3310 shown in FIG. 33.

[0300] The correction coefficient calculation unit 3610 pre-acquires the bit rate specified by the user. Further, the correction coefficient calculation unit 3610 acquires the re-bit rate of the re-encoded data generated by the re-encoding unit 450 and compares it with the bit rate specified by the user to calculate the correction coefficient α. Also, the correction coefficient calculation unit 3610 notifies the quantization value map generation unit 1420 of the calculated correction coefficient α.

[0301] Thereby, the quantization value map can be corrected so as to approach the bit rate specified by the user.

[0302] <Specific example of the processing of the correction coefficient calculation unit> Next, a specific example of the processing of the correction coefficient calculation unit 3610 will be described. FIG. 37 is the 5th diagram showing a specific example of the processing of the correction coefficient calculation unit. As shown in the example of FIG. 37, the correction coefficient calculation unit 3610 calculates the correction coefficient α based on the following formula (7).

[0303]

Equation

[0304] On the other hand, when the bit rate specified by the user is greater than the re-bit rate of the re-encoded data corresponding to the current image data, one or more correction coefficients α are calculated, so the quantization values in the corrected quantization value map become larger than before correction.

[0305] Note that in the above formula (7), the responsiveness is a parameter for making the ratio between the bit rate specified by the user and the re-bit rate of the re-encoded data corresponding to the current image data be gradually reflected without being directly reflected in the quantization value.

[0306] As shown in FIG. 37, the correction coefficient α calculated by the correction coefficient calculation unit 3610 is multiplied by the quantization value map 1510 generated by the quantization value map generation unit 1420, and the quantization value map 1510 is corrected to generate a corrected quantization value map 1520.

[0307] <Flow of Image Processing by Image Processing System> Next, the flow of image processing by the image processing system 100 according to the 13th embodiment will be described. FIG. 38 is a 13th flowchart showing the flow of image processing. The differences from FIG. 32 are steps S3801 and S3802.

[0308] In step S3801, the transcoding unit 121 of the server device 130 acquires the re-bit rate of the re-encoded data corresponding to the current image data.

[0309] In step S3802, the transcoding unit 121 of the server device 130 calculates the correction coefficient α using the bit rate specified by the user and the re-bit rate of the re-encoded data corresponding to the current image data.

[0310] As is clear from the above description, the image processing system 100 according to the 13th embodiment corrects the quantization value map generated based on the information regarding the region and the quantization value based on the bit rate specified by the user and the re-bit rate of the re-encoded data.

[0311] Thus, according to the image processing system 100 according to the 13th embodiment, the quantization value map can be corrected so that the re-bit rate of the re-encoded data approaches the user-specified bit rate.

[0312] As a result, according to the image processing system 100 according to the 13th embodiment, the same effects as those of the first embodiment can be enjoyed, and the occurrence of transmission delay can be avoided.

[0313] [Other Embodiments] In each of the above embodiments, the imaging device 110 and the hierarchical encoding device 111 have been described as separate devices, but the imaging device 110 and the hierarchical encoding device 111 may be integrated devices. Alternatively, the imaging device 110 may have some of the functions included in the hierarchical encoding device 111 and the image processing device 120.

[0314] Also, in each of the above embodiments, the compression information determination unit 310 has been described as being realized in the hierarchical encoding device 111, but the compression information determination unit 310 may be realized in, for example, the server device 130. In this case, based on the re-decoded data, information regarding the region and the quantization value is determined, and the determined information regarding the region and the quantization value is transmitted to the hierarchical encoding device 111, and thus is reflected in the encoding process of the next image data.

[0315] Also, in each of the above embodiments, the compression information determination unit 310 determines the limit quantization value by increasing the quantization value in a predetermined step width, but the method for determining the limit quantization value is not limited to this. For example, the compression information determination unit 310 may determine the limit quantization value by analyzing the recognition state and the recognition process by AI.

[0316] Also, in each of the above embodiments, the region separation unit 320 has been described as separating the image data of each frame included in the moving image data into first image data and second image data. However, the image data separated by the region separation unit 320 is not limited to two types, and may be three or more types. Note that when the image data is separated into three or more types, three or more types of encoded data will be generated.

[0317] Also, in each of the above embodiments, when the quantization value map generation unit 440 or the like generates a quantization value map based on information regarding the region and the quantization value, a limit quantization value or a quantization value close to the limit quantization value is set for the target region, and a predetermined quantization value is set for the non-target region. However, the method of setting the quantization value is not limited to this. For example, when the limit quantization values within the target region are not uniform, the quantization value map may be generated by uniformly setting the minimum quantization value or by uniformly setting the average quantization value.

[0318] Also, in each of the above embodiments, for a region that did not contain a recognition target when the video analysis unit 132 performed recognition processing by AI, a quantization value map may be generated by a generation method different from the generation method described in each of the above embodiments. For example, for a region that did not contain a recognition target, the quantization value map may be generated so that the data amount of the re-encoded data becomes smaller.

[0319] Also, the recognition processing by AI described in each of the above embodiments may include, in addition to deep learning processing, analysis processing or the like that obtains a result based on analysis by a computer or the like.

[0320] Also, in the tenth to twelfth embodiments above, the re-decoding unit 2710 is arranged in the transcoding unit 121, and it has been described that the transcoding unit 121 generates re-decoded data. However, the transcoding unit 121 may obtain the re-decoded data from, for example, the video analysis unit 132.

[0321] In the above-described first embodiment, it was mentioned that there is no need to incorporate a new function into the image recognition program of the server device 130. At this time, the image recognition program refers to, for example, · receiving encoded data for which the limit of what the AI can recognize the recognition target is not considered, · decrypting the received encoded data and performing video analysis, which refers to an application (i.e., a general application that receives encoded data and performs video analysis). That is, according to each of the above embodiments, it becomes possible to apply encoded data for which the limit of what the AI can recognize the recognition target is considered without changing the application.

[0322] Also, in the above-described sixth to eighth embodiments, the application area when generating the quantization value map using the MAD value or the PSNR value was not mentioned. However, for example, · only the area including the recognition target, or · only the area including the necessary recognition target among the recognition targets, or · only the area that is any of the above areas and has been narrowed down or expanded by an operation based on the requirements of the application, may be applied.

[0323] Also, in each of the above embodiments, a quantization value map considering the limit of what the AI can recognize the recognition target was described. However, depending on the use of video analysis in the server device 130, a quantization value map considering the limit of what the AI can recognize the recognition target as intended may be generated. Note that what is meant by the AI being able to recognize the recognition target as intended is, for example, in addition to the video analysis unit 132 being able to recognize the recognition target, it refers to decoded data with an image quality that minimizes the influence of quantization error and coding noise during the encoding process.

[0324] Note that the configurations and the like described in the above embodiments are not limited to the configurations shown here, such as combinations with other elements. Regarding these points, it is possible to make changes without departing from the spirit of the present invention, and they can be appropriately determined according to the application form.

Description of Symbols

[0325] 100: Image processing system 110: Imaging device 111: Hierarchical encoding device 121: Transcoding section 130: Server device 131: Re-encoded data acquisition section 132: Video analysis section 133: Video display section 310: Compression information determination section 320: Region separation section 330: First encoding section 340: Second encoding section 410: First decoding section 420: Second decoding section 430: Reconstruction section 440: Quantization value map generation section 450: Re-encoding section 710: Reconstruction section 720: Quantization value map generation section 1010: Compression information determination section 1020: First encoding section 1030: Second encoding section 1040: Reconstruction section 1050: Quantization value map generation section 1210: Bit rate acquisition section 1220: Quantization value map generation section 1410: Correction coefficient calculation section 1420: Quantization value map generation section 1710: Image MAD calculation section 1720: Reconstructed image MAD calculation section 1730: Quantization value calculation section 1740: Quantization value map generation section 2210: Quantization value calculation section 2220: Quantization value map generation section 2510: Minimum value calculation section 2710: Re-decoding section 2720: PSNR calculation unit 2730: Correction coefficient calculation unit 3010: Recognition unit 3020: Correction coefficient calculation unit 3310: Correction coefficient calculation unit 3610: Correction coefficient calculation unit

Claims

1. A determination unit that determines, based on the result of recognition processing, a target region necessary for recognizing a recognition target in image data and a non-target region outside the target region, and a quantization value of the target region and a quantization value of the non-target region necessary for recognizing the recognition target; A first encoding unit that encodes the entire region of the image data with the quantization value of the target region to generate first encoded data; A second encoding unit that encodes the entire region of the image data with the quantization value of the non-target region to generate second encoded data; A reconstruction unit that generates reconstructed image data using the target region in the first decoded data obtained by decoding the first encoded data and the non-target region in the second decoded data obtained by decoding the second encoded data; A re-encoding unit that re-encodes the reconstructed image data to generate re-encoded data An image processing system having the above.

2. The image processing system further includes a separation unit that separates the image data into image data with the non-target region as an invalid image and image data with the target region as an invalid image, The first encoding unit encodes the entire region of the image data with the non-target region as an invalid image with the quantization value of the target region, The second encoding unit encodes the entire region of the image data with the target region as an invalid image with the quantization value of the non-target region, The image processing system according to claim 1.

3. The image processing system further includes a generation unit that generates a quantization value map based on the quantization value of the target region and the quantization value of the non-target region, The re-encoding unit re-encodes the reconstructed image data using the generated quantization value map to generate re-encoded data, The image processing system according to claim 1.

4. The generation unit Based on information indicating the target region and information indicating the quantization value of the target region included in the first encoded data or transmitted in association with the first encoded data, Based on information indicating the non-target region and information indicating the quantization value of the non-target region included in the second encoded data or transmitted in association with the second encoded data, The image processing system according to claim 3, wherein the quantization value map is generated.

5. The reconstruction unit Specifies in advance a region that is determined to be an invalid region when the re-encoded data is re-decoded, and generates the reconstructed image data with the region as an invalid image. The image processing system according to claim 1.

6. The generation unit Identify in advance a region that is determined to be an invalid region when the re-encoded data is decoded again, and generate the quantization value map so that the quantization value of the region is maximized. The image processing system according to claim 3.

7. The generation unit Generate the quantization value map so as to realize a re-bit rate for transmitting the re-encoded data, which is determined based on the bit rate when transmitting the first encoded data and the second encoded data. The re-encoding unit re-encodes the reconstructed image data using the generated quantization value map to generate re-encoded data. The image processing system according to claim 3.

8. Further include a calculation unit that calculates a correction coefficient for correcting the quantization value map based on the bit rate when transmitting the first encoded data and the second encoded data and the re-bit rate when transmitting the re-encoded data. The generation unit Correct the quantization value map by multiplying the generated quantization value map by the correction coefficient. The image processing system according to claim 3.

9. Further include a generation unit that generates a quantization value map based on a value indicating an attribute of the image data and a value indicating an attribute of the reconstructed image data. The re-encoding unit re-encodes the reconstructed image data using the generated quantization value map to generate re-encoded data. The image processing system according to claim 1.

10. The generation unit Generate the quantization value map by using a quantization value calculated based on the difference between a value indicating an attribute of the image data and a value indicating an attribute of the reconstructed image data. The image processing system according to claim 9.

11. The generation unit When the image data is a P picture, generate the quantization value map after adjusting the quantization value according to the number of encoded blocks to which the intra prediction mode is applied in the first encoding unit. The image processing system according to claim 10.

12. Further include a generation unit that generates a quantization value map based on a value indicating an attribute of the reconstructed image data. The re-encoding unit re-encodes the reconstructed image data using the generated quantization value map to generate re-encoded data. The image processing system according to claim 1.

13. The relationship between the value indicating the attribute of the reconstructed image data and the re-bit rate when transmitting the re-encoded data is predetermined for each different quantization value. The generation unit refers to the relationship, and a quantization value map is generated by deriving a quantization value at which the re-bit rate corresponding to the value indicating the attribute of the reconstructed image data becomes the target bit rate. The image processing system according to claim 12.

14. The apparatus further includes a calculation unit that calculates a correction coefficient for correcting the quantization value map based on a change in PSNR calculated for the re-decoded data obtained by re-decoding the re-encoded data. The generation unit corrects the quantization value map by multiplying the generated quantization value map by the correction coefficient. The image processing system according to claim 3.

15. The apparatus further includes a calculation unit that calculates a correction coefficient for correcting the quantization value map based on a change in recognition rate when performing recognition processing on the re-decoded data obtained by re-decoding the re-encoded data. The generation unit corrects the quantization value map by multiplying the generated quantization value map by the correction coefficient. The image processing system according to claim 3.

16. The apparatus further includes a calculation unit that calculates a correction coefficient for correcting the quantization value map based on the ratio between the PSNR calculated for the re-decoded data obtained by re-decoding the re-encoded data and the specified PSNR. The generation unit corrects the quantization value map by multiplying the generated quantization value map by the correction coefficient. The image processing system according to claim 3.

17. The apparatus further includes a calculation unit that calculates a correction coefficient for correcting the quantization value map based on the ratio between the re-bit rate when transmitting the re-encoded data and the specified bit rate. The generation unit corrects the quantization value map by multiplying the generated quantization value map by the correction coefficient. The image processing system according to claim 3.

18. Based on the result of the recognition process, a target area necessary for recognizing a recognition target in the image data, a non-target area other than the target area, a quantization value of the target area necessary for recognizing the recognition target, and a quantization value of the non-target area are determined, and the entire area of the image data is acquired as first encoded data encoded with the quantization value of the target area and second encoded data encoded with the quantization value of the non-target area. An image processing apparatus, a reconstruction unit that generates reconstructed image data using the target area in the first decoded data obtained by decoding the first encoded data and the non-target area in the second decoded data obtained by decoding the second encoded data; a re-encoding unit that re-encodes the reconstructed image data to generate re-encoded data An image processing apparatus having the above. [

19. ] Based on the result of the recognition process, a target area necessary for recognizing a recognition target in the image data, a non-target area other than the target area, a quantization value of the target area necessary for recognizing the recognition target, and a quantization value of the non-target area are determined, and the entire area of the image data is acquired as first encoded data encoded with the quantization value of the target area and second encoded data encoded with the quantization value of the non-target area. A computer of an image processing apparatus, generates reconstructed image data using the target area in the first decoded data obtained by decoding the first encoded data and the non-target area in the second decoded data obtained by decoding the second encoded data, re-encodes the reconstructed image data to generate re-encoded data An image processing method in which a computer executes the process. [

20. ] Based on the result of the recognition process, a target area necessary for recognizing a recognition target in the image data, a non-target area other than the target area, a quantization value of the target area necessary for recognizing the recognition target, and a quantization value of the non-target area are determined, and the entire area of the image data is acquired as first encoded data encoded with the quantization value of the target area and second encoded data encoded with the quantization value of the non-target area. A computer of an image processing apparatus, generates reconstructed image data using the target area in the first decoded data obtained by decoding the first encoded data and the non-target area in the second decoded data obtained by decoding the second encoded data, re-encodes the reconstructed image data to generate re-encoded data An image processing program for causing a computer to execute processing.

Images