Information processing device, communication system, and information processing method

Abstract

Provided is an information processing device comprising: an acquisition unit that acquires a video captured by an imaging device; a generation unit that, on the basis of a first frame captured at a first time point by the imaging device and a second frame captured at a second time point after the first time point by the imaging device, generates map information indicating the degree of image quality corresponding to the pixel position in the first frame; and a compression unit that compresses the first frame on the basis of the map information.

Description

Information processing device, communication system, and information processing method

[0001] The present disclosure relates to an information processing device, a communication system, and an information processing method.

[0002] Patent Literature 1 discloses a technique for performing inter-frame compression of moving images. In this technique, the difference between a frame to be processed and a referenceable frame is calculated, and if the difference result is smaller than a threshold A, the frame is designated as a reference frame candidate, and motion prediction processing is performed only on the reference frame candidate, thereby reducing the amount of calculation required for the entire inter-frame compression processing.

[0003] JP 2009-147847 A

[0004] However, the technology described in Patent Document 1 leaves room for improvement in video compression, for example, in lossy compression (irreversible compression) that compresses data by reducing its content.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide a technology that can appropriately compress moving images.

[0006] In a first aspect of the present disclosure, there is provided an information processing device having an acquisition unit that acquires a video captured by a camera, a generation unit that generates map information indicating a degree of image quality according to pixel position in a first frame based on a first frame captured by the camera at a first time point and a second frame captured by the camera at a second time point later than the first time point, and a compression unit that compresses the first frame based on the map information.

[0007] In addition, a second aspect of the present disclosure provides an information processing method that acquires a video captured by a camera, generates map information indicating the degree of image quality according to pixel position in the first frame based on a first frame captured by the camera at a first time point and a second frame captured by the camera at a second time point later than the first time point, and compresses the first frame based on the map information.

[0008] In addition, in a third aspect of the present disclosure, a communication system is provided that includes an acquisition unit that acquires a video captured by a camera device, a generation unit that generates map information indicating a degree of image quality according to pixel positions in a first frame based on a first frame captured by the camera device at a first time point and a second frame captured by the camera device at a second time point later than the first time point, and a compression unit that compresses the first frame based on the map information, encodes a video including each compressed frame, and transmits the encoded video to the receiving terminal, wherein the receiving terminal decodes the video received from the information processing device.

[0009] According to one aspect, the video can be appropriately compressed.

[0010] 1 is a diagram showing an example of the configuration of an information processing device according to an embodiment; FIG. 2 is a diagram showing an example of the configuration of an information processing device according to an embodiment; FIG. 3 is a diagram showing an example of the configuration of a communication system according to an embodiment; FIG. 4 is a diagram showing an example of the hardware configuration of an information processing device and a receiving terminal according to an embodiment; FIG. 5 is a flowchart showing an example of learning processing of an information processing device according to an embodiment; FIG. 6 is a diagram showing an example of a moving image DB according to an embodiment; FIG. 7 is a flowchart showing an example of processing of an information processing device according to an embodiment; FIG. 8 is a diagram showing an example of processing of a generation unit and a compression unit according to an embodiment;

[0011] The principles of the present disclosure will be described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the disclosure. The disclosure described herein may be implemented in various ways other than those described below.

[0012] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. <Configuration> <<Configuration of Information Processing Device 10>> The configuration of the information processing device 10 according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 10 according to the embodiment. The information processing device 10 has an acquisition unit 11, a generation unit 12, and a compression unit 13. Each of these units may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as a processor and memory of the information processing device 10.

[0014] The acquisition unit 11 acquires a video captured by a camera device, the video including a first frame captured by the camera device at a first time point and a second frame captured by the camera device at a second time point after the first time point.

[0015] The generator 12 generates map information indicating the degree of image quality according to pixel positions in the first frame based on the first and second frames acquired by the acquirer 11. Note that the generator 12 may, for example, input the first and second frames to a trained model and output the map information. The compressor 13 compresses the first frame based on the map information generated by the generator 12.

[0016] <<Configuration of Information Processing Device 40>> Next, the configuration of the information processing device 40 according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 40 according to the embodiment. The information processing device 40 has an acquisition unit 41, a generation unit 42, and an evaluation unit 43. Each of these units may be realized by cooperation between one or more programs installed in the information processing device 40 and hardware such as a processor and memory of the information processing device 40.

[0017] The acquisition unit 41 acquires a video for machine learning. The generation unit 42 generates multiple pieces of map information corresponding to a third frame included in the video and a fourth frame captured at a time point later than the third frame. Furthermore, based on the evaluation result by the evaluation unit 43, the generation unit 42 generates a trained model for generating map information based on a first frame captured by an imaging device at a first time point and a second frame captured by the imaging device at a second time point later than the first time point.

[0018] The evaluation unit 43 compresses each third frame based on each of the multiple pieces of map information generated by the generation unit 42. Then, the evaluation unit 43 evaluates the compression quality when the video including each compressed third frame is encoded by a specific encoding method using inter-frame prediction.

[0019] Next, the configuration of a communication system 1 according to an embodiment will be described with reference to FIG. 3 . <System Configuration> FIG. 3 is a diagram illustrating an example of the configuration of a communication system 1 according to an embodiment. In the example of FIG. 3 , the communication system 1 includes an information processing device 10, an image capturing device 20, a receiving terminal 30 (an example of a "decoding means"), and an information processing device 40. Note that the number of information processing devices 10, image capturing devices 20, receiving terminals 30, and information processing devices 40 is not limited to the example of FIG. 3 . Note that the technology of the present disclosure can be used in various services that encode and transmit video, such as the distribution of stored video, the compression of video for recording (saving) on ​​a recording medium, and the distribution of video in real time. In this case, the technology of the present disclosure can also be used, for example, in remote medical consultations in which video captured by a camera (image capturing device) such as an endoscope is distributed to a remote specialist and instructions are received from the specialist.

[0020] In the example of Fig. 3, the information processing device 10, the receiving terminal 30, and the information processing device 40 are connected so as to be able to communicate via a network N. Examples of the network N include, for example, the Internet, a mobile communication system, a wireless LAN (Local Area Network), short-range wireless communication such as BLE, a LAN, and a bus. Examples of the mobile communication system include, for example, a fifth-generation mobile communication system (5G), a fourth-generation mobile communication system (4G), and a third-generation mobile communication system (3G). Also, in the example of Fig. 3, the information processing device 10 and the imaging device 20 are connected by a cable. Note that the information processing device 10 and the imaging device 20 may be connected so as to be able to communicate via a wired or wireless connection via, for example, the network N.

[0021] The imaging device 20 captures a moving image and outputs the captured moving image to the information processing device 10. The imaging device 20 may be built into the housing of the information processing device 10, or may be externally connected to the information processing device 10.

[0022] The information processing device 10 may be, for example, a personal computer, a smartphone, a server, a cloud server, etc. The information processing device 10 compresses the video captured by the imaging device 20 using the trained model generated by the information processing device 40. The information processing device 10 also transmits the compressed video to the receiving terminal 30.

[0023] The receiving terminal 30 may be, for example, a device such as a personal computer, a smartphone, a server, a cloud server, etc. The receiving terminal 30 decodes the video received from the information processing device 10 and plays (displays) it.

[0024] The information processing device 40 may be, for example, a device such as a cloud server or a server. The information processing device 40 generates a trained model for generating map information used when compressing a video using machine learning, and distributes the trained model to the information processing device 10. Note that the map information in the present disclosure is information indicating the level of image quality (level of compression) according to the pixel position in a frame. Note that the map information may be information indicating the level of image quality for each pixel, or may be information indicating the level of image quality for each of multiple pixels (e.g., a 4x4 block).

[0025] <Hardware Configuration> Fig. 4 is a diagram showing an example of the hardware configuration of the information processing devices 10, 40 and the receiving terminal 30 according to the embodiment. In the example of Fig. 4, the information processing devices 10, 40 and the receiving terminal 30 (computer 100) include a processor 101, a memory 102, and a communication interface 103. These components may be connected via a bus or the like. The memory 102 stores at least a part of the program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0026] When the program 104 is executed by the processor 101, memory 102, and other components in cooperation with each other, the computer 100 performs at least some of the processing of the embodiments of the present disclosure. The memory 102 may be of any type. As a non-limiting example, the memory 102 may be a non-transitory computer-readable storage medium. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Although only one memory 102 is shown in the computer 100, several physically different memory modules may be present in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and, as a non-limiting example, a processor based on a multi-core processor architecture. The computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-slaved to a clock that synchronizes the main processor.

[0027] Embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.

[0028] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor or device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided among program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

[0029] The program code for executing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus. When the program code is executed by the processor or controller, the functions / acts in the flowcharts and / or implementing block diagrams are performed. The program code may be executed entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0030] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0031] <Processing> <<Learning Phase>> Next, an example of a learning process (process for generating a trained model) of the information processing device 40 according to the embodiment will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a flowchart showing an example of the learning process of the information processing device 40 according to the embodiment. Fig. 6 is a diagram showing an example of a video DB (Data Base) 601 according to the embodiment.

[0032] In step S101, the acquisition unit 41 of the information processing device 40 acquires videos and the like for machine learning from the video DB 601. The video DB 601 stores, in advance, videos and the like that have been previously captured by the imaging device 20. Note that the video DB 601 may be stored, for example, in a storage device inside the information processing device 10 or in a storage device external to the information processing device 10.

[0033] In the example of FIG. 6 , the video DB 601 stores still images, playback times, and movement amounts in association with pairs of video IDs and frame IDs. The video ID is identification information for the video. The frame ID is identification information for a frame (still image) in the video. The still image is image data for the still image of the frame that constitutes the video. The playback time is, for example, information indicating the playback time from the beginning of the video to the current frame. The movement amount may be, for example, information indicating the three-dimensional movement amount of the image capture device 20 from the time the previous frame to the current frame was captured to the time the current frame was captured. The movement amount may be, for example, data calculated based on data measured by an inertial measurement unit (IMU) included in the image capture device 20. Note that the IMU may be a sensor that detects three-dimensional inertial movement (translational movement and rotational movement in three orthogonal axial directions) by, for example, detecting translational movement using an acceleration sensor and rotational movement using an angular velocity (gyro) sensor.

[0034] Next, the generation unit 42 of the information processing device 40 selects a specific frame (third frame) from among the frames included in the moving image (step S102). Here, the generation unit 42 may, for example, select a frame associated with a randomly determined playback time from among the frames included in the moving image as the third frame. Alternatively, the generation unit 42 may, for example, select a frame designated by an operator (administrator) of the information processing device 40 as the third frame.

[0035] Next, the generation unit 42 of the information processing device 40 generates multiple pieces of map information corresponding to the third frame and one or more frames (fourth frames) captured at a time later than the third frame (step S103). Here, the generation unit 42 may generate the map information using a generation network (generator) that is an AI (artificial intelligence) that receives, for example, a certain frame and one or more frames captured at a time later than the certain frame, and generates (outputs) map information using the differences between the frames. In this case, the generation unit 42 may use any AI technology as the generation network. The generation network may be, for example, a generation AI that uses a convolutional neural network (CNN). In this case, the generation network may be, for example, a generation AI that uses a residual network (ResNet).

[0036] The generator 42 may use a generation network to generate a plurality of combinations of map information and estimated (inferred) values ​​of compression quality when the map information is used, and may output a specific number of map information in descending order of the estimated quality values.

[0037] The generator 42 may generate multiple pieces of map information using the third frame and a fourth frame that includes only one frame. Alternatively, the generator 12 may generate map information using the third frame and a fourth frame that includes multiple frames. In this case, the fourth frame may include frames captured at multiple times at specific intervals (e.g., 1 / 60 second intervals) after the third frame was captured.

[0038] (Example of machine learning using the amount of movement detected by the sensor of the imaging device 20) The generator 12 may generate map information using the amount of movement detected by the sensor of the imaging device 20 in addition to the third and fourth frames. This allows learning of more appropriate map information according to the amount of movement of the imaging device 20, for example, in the case of an endoscopic examination or video distribution on a smartphone. In this case, the generator 42 may generate multiple pieces of map information based on the third frame, the fourth frame, and the amount of movement when the third frame was captured.

[0039] Next, the evaluation unit 43 of the information processing device 40 compresses the third frame based on each of the multiple pieces of map information generated by the generation unit 42 (step S104). Here, the evaluation unit 43 may compress the data size by blurring pixel values ​​at each position of the third frame specified in the map information based on the level of image quality specified in the map information. For example, when a specific frame is encoded using an encoding method (video compression method) that uses inter-frame prediction, some of the pixels restored from the compressed third frame will be reused in the fourth frame by motion compensation or the like.

[0040] The ratio of the amount of data to the image quality is improved by varying the image quality of the compressed third frame depending on the degree of reuse (e.g., how small the residual error, which depends on the accuracy of motion compensation, is). Note that the residual error may be, for example, the difference between the second pixel in the fourth frame generated by reusing the first pixel in the third frame using motion compensation or the like, and the second pixel in the captured fourth frame.

[0041] According to the present disclosure, it is expected that the degree of reuse can be estimated as map information, and since the map information is used for compression, it is expected that the ratio of data volume to image quality can be improved, which makes it possible to deliver video with higher image quality using a smaller communication bandwidth, for example.

[0042] Motion compensation is, for example, a type of inter-frame prediction technology, which detects the direction and speed of movement of a subject moving within a video based on other frames, and predicts the current frame using an image shifted by the amount of movement.

[0043] In this case, the evaluation unit 43 may, for example, change the pixel value of a pixel to a value closer to the pixel values ​​of each pixel surrounding (neighboring) the pixel, the lower the level of image quality specified in the map information for that pixel (the higher the level of compression).

[0044] The evaluation unit 43 may also compress the image of each region of the third frame by reducing the image based on each piece of map information. In this case, each piece of map information may include, for example, information indicating the position and size of each rectangular (including square) region in the third frame and the number of pixels in each rectangular region after compression.

[0045] Next, the evaluation unit 43 of the information processing device 40 evaluates the compression quality of each video including each compressed third frame when it is encoded using a specific encoding method using inter-frame prediction (step S105). Here, the evaluation unit 43 may calculate a value indicating the compression quality as an evaluation value, for example, based on at least one of a high compression rate and low image quality degradation. In this case, the evaluation unit 43 may determine a higher value indicating the compression quality, for example, the higher the compression rate and the lower the image quality degradation when the third frame is encoded using inter-frame prediction. Note that the image quality degradation may be calculated based on, for example, the magnitude of the difference between the image before encoding the third frame and the image decoded after encoding the third frame using inter-frame prediction.

[0046] The specific encoding method may be, for example, High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), etc. Furthermore, the specific encoding method may be, for example, any known video compression method that uses AI technology or the like in inter-frame prediction or the like.

[0047] Inter-frame prediction is a technique for predicting a current frame from at least one of a previous frame and a subsequent frame. In a coding method using inter-frame prediction, a frame coded without using inter-frame prediction is called an I-frame (Intra-coded Frame), an intra-frame, or a key frame. A frame coded using only forward prediction is called a P-frame (Predicted Frame). A frame coded by selecting one of forward prediction, backward prediction, and bidirectional prediction is called a B-frame (Bi-directional Predicted Frame). Forward prediction is a method of predicting from a previous frame (e.g., the previous frame). Backward prediction is a method of predicting from a subsequent frame (e.g., the subsequent frame). Bidirectional prediction is a method of predicting from previous and subsequent frames.

[0048] For example, if the evaluation unit 43 reduces the image of each region of the third frame based on the map information in the process of step S104, the evaluation unit 43 may encode and decode the third frame, and then enlarge the image of each region of the third frame based on the map information, thereby restoring the third frame to the same pixel size as before compression.

[0049] Next, the generation unit 42 of the information processing device 40 performs machine learning based on the evaluation result by the evaluation unit 43 (step S106). Here, the generation unit 42 may perform machine learning on the generation network, for example, using the evaluation value for each piece of map information calculated by the evaluation unit 43 as correct answer data.

[0050] Next, the generation unit 42 of the information processing device 40 determines whether the condition for terminating the machine learning has been satisfied (step S107). The generation unit 42 may determine that the condition for terminating the machine learning has been satisfied, for example, when the number of times of machine learning in step S106 is equal to or greater than a threshold. Alternatively, the generation unit 42 may determine that the condition for terminating the machine learning has been satisfied, for example, when the evaluation by the evaluation unit 43 is equal to or greater than a threshold.

[0051] If the conditions for terminating machine learning are not met (NO in step S107), the process proceeds to step S102. On the other hand, if the conditions for terminating machine learning are met (YES in step S107), the trained model is output (generated) (step S108), and the process ends. The generated trained model is delivered to the information processing device 10 at an appropriate timing.

[0052] <<Inference Phase>> Next, an example of processing by the information processing device 10 according to the embodiment will be described with reference to FIG. 7 . FIG. 7 is a flowchart showing an example of processing by the information processing device 10 according to the embodiment. Note that the processing from step S202 to step S203 may be executed for each frame other than the final frame of the moving image. Also, the processing from step S202 to step S203 may be executed by an external device separate from the information processing device 10. Note that the generation unit 12 may execute the processing of FIG. 7 when, for example, the difference (residual) between each pixel in a second frame generated by reusing each pixel in a first frame and each pixel in the captured second frame is equal to or less than a threshold.

[0053] In step S201 , the acquisition unit 11 of the information processing device 10 acquires a video or the like captured by the imaging device 20 .

[0054] Next, the generation unit 12 of the information processing device 10 generates map information indicating the degree of image quality corresponding to pixel positions in a frame (first frame) captured at a certain time point (first time point) and one or more frames (second frames) captured at a second time point after the first time point (step S202). Here, the generation unit 12 may generate one piece of map information based on the first frame and the second frame using a generative network that uses a trained model generated by the information processing device 40, for example, by processing similar to step S103 of FIG. 5. In this case, the generation unit 12 may, for example, use the generative network to generate multiple combinations of map information and estimated (inferred) values ​​of compression quality when the map information is used. Then, the generation unit 12 may output (select) the map information with the highest estimated value of quality.

[0055] The generator 12 may generate map information using a first frame and a second frame including only one frame. Alternatively, the generator 12 may generate map information using the first frame and a second frame including multiple frames. In this case, the second frame may include frames captured at multiple time points at specific intervals (e.g., 1 / 60 second intervals) after the first time point.

[0056] (Example of Modifying Map Information Using a Region of Interest (ROI)) The generation unit 12 may modify the map information for the first frame using a region of interest in the first frame. This may improve the image quality of, for example, an affected area during an endoscopic examination or a facial area when distributing a video of a person. In this case, the generation unit 12 may, for example, use AI or the like to recognize a specific object from the first frame and detect the area of ​​the recognized object in the first frame as the region of interest. Then, the generation unit 12 may, for example, increase a value indicating the level of image quality of the region of interest in the map information. In this case, the generation unit 12 may, for example, set the value indicating the level of image quality of the region of interest in the map information to the highest value (e.g., 100) so as not to blur the region of interest.

[0057] (Example of Generating Map Information Using the Amount of Movement Detected by a Sensor of the Image Capturing Device 20) The generator 12 may generate map information using not only the first frame and the second frame, but also the amount of movement detected by a sensor of the image capturing device 20 when the first frame was captured. This allows, for example, endoscopic examinations or video distribution via smartphones to generate more appropriate map information according to the amount of movement of the image capturing device 20. In this case, the generator 12 may generate a piece of map information based on the first frame, the second frame, and the amount of movement when the first frame was captured using a generative network using a trained model generated by the information processing device 40. In this case, the generator 12 may, for example, use the generative network to generate multiple combinations of map information and estimated (inferred) values ​​of compression quality when the map information is used. Then, the generator 12 may output (select) the map information with the highest estimated quality value. Note that the amount of movement may be calculated based on data measured by an IMU (Integrated Memory Unit) included in the image capturing device 20, for example.

[0058] Next, the compression unit 13 of the information processing device 10 compresses the first frame based on the map information generated by the generation unit 12 (step S203). Here, the compression unit 13 may compress the first frame, for example, by processing similar to step S104 in Fig. 5. In this case, the compression unit 13 may compress the data size by, for example, blurring pixel values ​​at each position in the first frame specified in the map information based on the level of image quality specified in the map information.

[0059] For example, when encoding a specific frame using an encoding method (video compression method) that uses inter-frame prediction, some pixels restored from the compressed first frame are reused in the second frame by motion compensation or the like. The ratio of image quality to data volume is improved by varying the image quality of the compressed first frame depending on the degree of reuse (e.g., how little residual error, which depends on the accuracy of motion compensation, is left). Note that the residual error may be, for example, the difference between a second pixel in a second frame generated by reusing a first pixel in the first frame using motion compensation or the like, and the second pixel in the captured second frame.

[0060] (Example of Compression by Reducing Pixel Size of Frame) The compression unit 13 may reduce the image of each region of the first frame based on, for example, map information. In this case, the map information may include, for example, information indicating the position and size of each rectangular (including square) region in the first frame and the number of pixels in each rectangular region after compression.

[0061] Next, the compression unit 13 of the information processing device 10 encodes the video having each frame including the first frame compressed based on the map information using a specific encoding method that uses inter-frame prediction (step S204). In this case, the specific encoding method may be, for example, High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), etc. Alternatively, the specific encoding method may be, for example, any known encoding method that uses AI technology or the like in inter-frame prediction or the like.

[0062] Next, the compression unit 13 of the information processing device 10 outputs the encoded video (step S205). Here, the compression unit 13 may, for example, transmit (distribute) the encoded data to the receiving terminal 30. The compression unit 13 may also, for example, record and save the encoded data on a specific recording medium. As a result, data obtained by encoding the video captured by the imaging device 20 is output.

[0063] In step S203, if the image of each region of the first frame is reduced based on the map information, the map information may be output (e.g., transmitted to the receiving terminal 30) in association with the first frame. Then, when decoding a video including the first frame, a decoding device such as the receiving terminal 30 may decode the first frame and then enlarge the image of each region of the first frame based on the map information. This allows the first frame to be restored to the same pixel size as before compression.

[0064] 8 is a diagram illustrating an example of the processing of the generating unit 12 and the compressing unit 13 according to the embodiment. In the example of FIG. 8, the generating unit 12 generates an I-frame x ０ and x ０ The subsequent frame x of１ and based on I-frame x ０ Then, the compression unit 13 generates map information (Utility Map) for the I frame x using the map information. ０ The frame x is encoded and then decoded. ０ ～ The map information is used to generate the I-frame x ０ The encoded data is transmitted to the receiving terminal 30.

[0065] In the example of FIG. 8, the generation unit 12 generates a P frame x １ and x １ The subsequent frame x of ２ and the residual １ In the example of FIG. 8, the residual is generated for the P frame x １ And frame x ０ ～ and P Frame x １ The map information is used to encode the difference frame 812 and then decode it to generate a compensated frame 811, which is a difference frame 812 between the difference frame 811 and the P frame x. The compression unit 13 then encodes the difference frame 812 using the map information and then decodes it to generate a frame 813. １ The data obtained by encoding the difference frame 813 using motion compensation is transmitted to the receiving terminal 30. At the receiving terminal 30, the frame x1 ～ is generated.

[0066] In the example of FIG. 8, the compression unit 13 generates a frame x1 ～ and P frame x １ By the same process as above, P frame x ２ The compression process is performed on the

[0067] <Others> In related technologies, for example, after the first frame (e.g., an I-frame) of a GOP (Group of Pictures) or a frame that is unclear (blurred) due to motion blur, it may take several frames for the compression performance of image quality and bit rate to converge appropriately. Therefore, appropriate compression performance may not be obtained in dynamic scenes. Note that, for example, during endoscopic examinations, the camera is moved as needed by the doctor or other operator, resulting in frequent motion blur. Therefore, it is considered more important that the time required for compression performance to converge is short.

[0068] On the other hand, according to the present disclosure, for example, the image quality of each region of the currently encoded frame after encoding is determined by AI or the like depending on the amount of pixel information reused in subsequent frames. If the amount of pixel information reused in subsequent frames (reuse rate) is high, a higher image quality or bit rate can be assigned to the currently encoded frame. Therefore, even in dynamic scenes, the time required for compression performance to converge is further shortened. Furthermore, the image quality can be improved while reducing the bit rate of the entire video. Therefore, the video can be appropriately compressed.

[0069] <Modifications> The information processing device 10, the receiving terminal 30, and the information processing device 40 may each be devices included in a single housing, but the information processing device 10, the receiving terminal 30, and the information processing device 40 of the present disclosure are not limited to this. Each unit of the information processing device 10, the receiving terminal 30, and the information processing device 40 may be realized, for example, by cloud computing configured with one or more computers. At least a portion of the processing of the information processing device 10 may be executed, for example, by the information processing device 40. At least a portion of the processing of the information processing device 40 may be executed, for example, by the information processing device 10. At least a portion of the processing of the receiving terminal 30 may be executed, for example, by the information processing device 10. The information processing device 10, the receiving terminal 30, and the information processing device 40 may be an integrated device. Such information processing devices 10, the receiving terminal 30, and the information processing device 40 are also included in examples of the "information processing device" of the present disclosure.

[0070] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0071] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) An information processing device comprising: an acquisition unit that acquires a video captured by a camera; a generation unit that generates map information indicating a degree of image quality corresponding to pixel positions in a first frame, based on a first frame captured by the camera at a first time point and a second frame captured by the camera at a second time point later than the first time point; and a compression unit that compresses the first frame based on the map information. (Supplementary Note 2) The generation unit inputs the first frame and the second frame to a trained model to output the map information, and the trained model generates multiple pieces of map information corresponding to a third frame included in a video and a fourth frame captured at a time point later than the third frame, and is generated by machine learning based on compression quality when the third frame is compressed based on each piece of generated map information. (Supplementary Note 3) The information processing device according to Supplementary Note 2, wherein the quality of the compression includes at least one of a high compression rate and low image quality degradation. (Supplementary Note 4) The information processing device according to Supplementary Note 1 or 2, wherein the compression unit compresses each frame included in the video based on each piece of map information, and then encodes the frame using a coding method using inter-frame prediction. (Supplementary Note 5) The information processing device according to Supplementary Note 1 or 2, wherein the compression unit reduces an image of each region of the first frame based on the map information, and outputs the map information associated with the first frame. (Supplementary Note 6) The information processing device according to Supplementary Note 1 or 2, wherein the second time point includes a plurality of time points, and the second frames include frames captured at each of the plurality of time points. (Supplementary Note 7) The information processing device according to Supplementary Note 1 or 2, wherein the generation unit modifies the map information using a region of interest in the first frame. (Supplementary Note 8) The information processing device according to Supplementary Note 1 or 2, wherein the generation unit generates the map information based on the first frame, the second frame, and an amount of movement when the first frame was captured.(Supplementary Note 9) An information processing device according to Supplementary Note 1, comprising: a second acquisition unit that acquires a video for machine learning; a second generation unit that generates a plurality of map information corresponding to a third frame included in the video and a fourth frame captured at a time later than the third frame; and an evaluation unit that compresses the third frame based on each of the plurality of map information and evaluates the compression quality when the video including the third frame after each compression is coded, wherein the second generation unit generates a trained model for generating map information based on a first frame captured by a camera device at a first time point and a second frame captured by the camera device at a second time point later than the first time point, based on an evaluation result by the evaluation unit; and the evaluation unit compresses the third frame based on each of the plurality of map information and evaluates the compression quality when the video including the third frame after each compression is coded using a specific coding method using inter-frame prediction. (Supplementary Note 10) The information processing device according to Supplementary Note 9, wherein the evaluation unit reduces an image of each region of the third frame based on each of the plurality of map information, and evaluates compression quality when the video including the reduced third frame is encoded by a specific encoding method using inter-frame prediction. (Supplementary Note 11) The information processing device according to Supplementary Note 9, wherein the fourth frame includes frames captured at each of a plurality of time points after the third frame. (Supplementary Note 12) The information processing device according to Supplementary Note 9, wherein the generation unit generates the plurality of map information based on the third frame, the fourth frame, and an amount of movement when the third frame was captured. (Supplementary Note 13) A communication system including: an acquisition means for acquiring a video captured by a photographing device; a generation means for generating map information indicating a degree of image quality according to pixel positions in a first frame based on a first frame captured by the photographing device at a first time point and a second frame captured by the photographing device at a second time point after the first time point; a compression means for compressing the first frames based on the map information, encoding and outputting a video including each compressed frame; and a decoding means for decoding the video encoded by the compression means.(Supplementary Note 14) The communication system according to Supplementary Note 13, wherein the generation means inputs the first frame and the second frame into a trained model to output the map information, and the trained model generates a plurality of map information corresponding to a third frame included in a video and a fourth frame captured at a point in time after the third frame, and is generated by machine learning based on compression quality when the third frame is compressed based on each generated map information. (Supplementary Note 15) The communication system according to Supplementary Note 14, wherein the compression quality includes at least one of a high compression rate and low image quality degradation. (Supplementary Note 16) The communication system according to Supplementary Note 13 or 14, wherein the compression means compresses each frame included in the video based on each map information, and then encodes the frame using a coding method that uses inter-frame prediction. (Supplementary Note 17) The communication system according to Supplementary Note 13, wherein the compression means reduces the image of each region of the first frame based on the map information and outputs the map information associated with the first frame, and the decoding means enlarges the image of each region of the first frame based on the map information. (Supplementary Note 18) An information processing method comprising: acquiring a video captured by an imaging device; generating map information indicating a degree of image quality corresponding to pixel positions in a first frame based on a first frame captured by the imaging device at a first time point and a second frame captured by the imaging device at a second time point later than the first time point; and compressing the first frame based on the map information. (Supplementary Note 19) The generating process inputs the first frame and the second frame to a trained model to output the map information, and the trained model generates a plurality of map information corresponding to a third frame included in the video and a fourth frame captured at a time point later than the third frame, and generates the map information by machine learning based on compression quality when the third frame is compressed based on each of the generated map information. (Supplementary Note 20) The information processing method of Supplementary Note 19, wherein the compression quality includes at least one of a high compression rate and low image quality degradation.and a generating unit configured to generate a plurality of pieces of map information corresponding to a third frame included in the video and a fourth frame captured at a time later than the third frame, and to evaluate a quality of compression when the video including the third frame after each compression is encoded, wherein the generating unit generates a trained model for generating map information based on the first frame captured by the video at the first time point and the second frame captured by the video at a second time point later than the first time point, based on the evaluation result by the evaluating unit. (Supplementary Note 23) An information processing method comprising: acquiring a video for machine learning; generating a plurality of map information corresponding to a third frame included in the video and a fourth frame captured at a time later than the third frame; compressing each of the third frames based on each of the plurality of map information; evaluating the quality of compression when encoding the video including the third frame after each compression; and generating, based on the evaluation results, a trained model for generating map information based on a first frame captured by a camera at a first time point and a second frame captured by the camera at a second time point later than the first time point.(Supplementary Note 24) A non-transitory computer-readable medium storing a program that causes a computer to execute the following processes: acquiring a video for machine learning; generating a plurality of map information corresponding to a third frame included in the video and a fourth frame captured at a time later than the third frame; compressing the third frame based on each of the plurality of map information; evaluating the compression quality when encoding the video including the third frame after each compression; and generating a trained model for generating map information based on the evaluation results, based on a first frame captured by a camera at a first time point and a second frame captured by the camera at a second time point later than the first time point.

[0072] REFERENCE SIGNS LIST 1 communication system 10 information processing device 11 acquisition unit 12 generation unit 13 compression unit 20 imaging device 30 receiving terminal 40 information processing device 41 acquisition unit 42 generation unit 43 evaluation unit

Claims

1. An information processing device having: an acquisition unit that acquires a video captured by a photographing device; a generation unit that generates map information indicating a degree of image quality according to pixel position in a first frame based on a first frame captured by the photographing device at a first time point and a second frame captured by the photographing device at a second time point later than the first time point; and a compression unit that compresses the first frame based on the map information.

2. The information processing device of claim 1, wherein the generation unit inputs the first frame and the second frame into a trained model to output the map information, and the trained model generates a plurality of map information corresponding to a third frame included in a video and a fourth frame captured at a point in time after the third frame, and is generated by machine learning based on the compression quality when the third frame is compressed based on each of the generated map information.

3. The information processing device according to claim 2, wherein the compression quality includes at least one of a high compression ratio and low degradation of image quality.

4. The information processing device according to claim 1 or 2, wherein the compression unit compresses each frame included in the video based on each map information, and then encodes the compressed frame using an encoding method that employs inter-frame prediction.

5. The information processing device according to claim 1 or 2, wherein the compression section reduces the image of each area of ​​the first frame based on the map information, and outputs the map information associated with the first frame.

6. An information processing device according to claim 1 or 2, wherein the second time point includes a plurality of time points, and the second frames include frames captured at each of the plurality of time points.

7. The information processing device according to claim 1 or 2, wherein the generation unit modifies the map information using a region of interest in the first frame.

8. The information processing device according to claim 1 or 2, wherein the generation unit generates the map information based on the first frame, the second frame, and an amount of movement when the first frame was captured.

9. An information processing device as described in claim 1, comprising: a second acquisition unit that acquires a video for machine learning; a second generation unit that generates multiple pieces of map information corresponding to a third frame included in the video and a fourth frame captured at a time later than the third frame; and an evaluation unit that compresses the third frame based on each of the multiple pieces of map information and evaluates the compression quality when the video including the third frame after each compression is encoded, wherein the second generation unit generates a trained model for generating map information based on a first frame captured by a shooting device at a first time point and a second frame captured by the shooting device at a second time point later than the first time point, based on an evaluation result by the evaluation unit; and the evaluation unit compresses the third frame based on each of the multiple pieces of map information and evaluates the compression quality when the video including the third frame after each compression is encoded using a specific encoding method using inter-frame prediction.

10. The information processing device according to claim 9, wherein the evaluation unit reduces the image of each region of the third frame based on each of the multiple map information, and evaluates the compression quality of the video including the third frame after each reduction when encoded using a specific encoding method that uses inter-frame prediction.

11. The information processing device according to claim 9, wherein the fourth frame includes frames captured at each of a plurality of time points after the third frame.

12. The information processing device according to claim 9, wherein the generation unit generates the plurality of pieces of map information based on the third frame, the fourth frame, and an amount of movement when the third frame was captured.

13. A communication system comprising: an acquisition means for acquiring a video captured by an image capture device; a generation means for generating map information indicating a degree of image quality corresponding to pixel positions in a first frame based on a first frame captured by the image capture device at a first time point and a second frame captured by the image capture device at a second time point later than the first time point; a compression means for compressing the first frames based on the map information, encoding and outputting a video including each compressed frame; and a decoding means for decoding the video encoded by the compression means.

14. The communication system described in claim 13, wherein the generation means inputs the first frame and the second frame into a trained model to output the map information, and the trained model generates a plurality of map information corresponding to a third frame included in a video and a fourth frame captured at a point in time after the third frame, and the map information is generated by machine learning based on the compression quality when the third frame is compressed based on each of the generated map information.

15. The communication system according to claim 14, wherein the compression quality includes at least one of a high compression rate and low image quality degradation.

16. A communication system according to claim 13 or 14, wherein said compression means compresses each frame included in said video based on each map information, and then encodes the compressed frame using an encoding method that employs inter-frame prediction.

17. The communication system of claim 13, wherein the compression means reduces the image of each area of ​​the first frame based on the map information and outputs the map information associated with the first frame, and the decoding means enlarges the image of each area of ​​the first frame based on the map information.

18. An information processing method comprising: acquiring a video captured by an imaging device; generating map information indicating a degree of image quality according to pixel position in the first frame based on a first frame captured by the imaging device at a first time point and a second frame captured by the imaging device at a second time point later than the first time point; and compressing the first frame based on the map information.

19. The information processing method of claim 18, wherein the generating process inputs the first frame and the second frame into a trained model to output the map information, and the trained model generates a plurality of map information corresponding to a third frame included in the video and a fourth frame captured at a point in time after the third frame, and the map information is generated by machine learning based on the compression quality when the third frame is compressed based on each of the generated map information.

20. The information processing method according to claim 19, wherein the compression quality includes at least one of a high compression rate and low degradation of image quality.

Images