Information processing apparatus, information processing method, information processing system, and program

Abstract

Provided is an information processing apparatus comprising: a calculation unit that calculates an amount of change of each pixel of a second frame relative to peripheral pixels of the pixel, the second frame being based on a frame at a point in time before a first frame in a moving image; and a compression unit that compresses, with a code amount corresponding to the amount of change at a position indicated by a motion vector estimated on the basis of the first frame and the second frame and corresponding to each of one or more pixels of the second frame, the motion vector. This enables appropriate compression of a motion vector used for compression of a moving image.

Description

Information processing device, information processing method, information processing system, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing system.

[0002] Patent Literature 1 discloses a technique for improving the accuracy of predictor vector candidates for blocks adjacent in the temporal direction. In Patent Literature 1, for a first coordinate in a block to be processed, a plurality of blocks including a block closest to the first coordinate in a picture adjacent in the temporal direction are determined, and at least one motion vector is selected from the motion vectors of the determined plurality of blocks.

[0003] JP 2013-005169 A

[0004] However, the technique described in Patent Document 1 does not consider the problem that occurs when the data size of a motion vector increases in order to improve the accuracy of the motion vector, for example.

[0005] In view of the above-mentioned problems, an object of the present disclosure is to provide a technology that can appropriately compress motion vectors used in video compression.

[0006] In a first aspect of the present disclosure, an information processing device is provided that has: a calculation unit that calculates the amount of change in each pixel of a second frame based on a frame that is earlier than a first frame in a moving image relative to surrounding pixels; and a compression unit that compresses the motion vector with a code amount that corresponds to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame.

[0007] In addition, a second aspect of the present disclosure provides an information processing method that calculates the amount of change relative to surrounding pixels of each pixel of a second frame based on a frame that is earlier than a first frame in a moving image, and compresses the motion vector with a code amount corresponding to the amount of change at a position indicated by the motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame.

[0008] In addition, a third aspect of the present disclosure provides an information processing device having an acquisition unit that acquires information indicating the amount of change relative to surrounding pixels of each pixel of a second frame based on a frame that is earlier than a first frame in a moving image, and a motion vector estimated based on the first frame and the second frame and compressed with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, and a restoration unit that restores the compressed motion vector acquired by the acquisition unit based on the amount of change.

[0009] In addition, a fourth aspect of the present disclosure provides an information processing method that acquires information indicating the amount of change relative to surrounding pixels of each pixel of a second frame based on a frame that is earlier than a first frame in a moving image, and a motion vector estimated based on the first frame and the second frame and compressed with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, and restores the compressed motion vector based on the amount of change.

[0010] In addition, in a fifth aspect of the present disclosure, there is provided an information processing system having a first information processing device and a second information processing device, wherein the first information processing device has a calculation unit that calculates an amount of change relative to surrounding pixels of each pixel of a second frame based on a frame that is earlier than the first frame in a moving image, and a compression unit that compresses the motion vector with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame, and the second information processing device has an acquisition unit that acquires the motion vector compressed with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, and a restoration unit that restores the compressed motion vector acquired by the acquisition unit based on the amount of change.

[0011] According to one aspect, motion vectors used in video compression can be appropriately compressed.

[0012] FIG. 1 is a diagram showing an example of the configuration of an information processing device that compresses moving images according to an embodiment. FIG. 2 is a diagram showing an example of the configuration of an information processing device that restores moving images according to an embodiment. FIG. 3 is a diagram showing an example of the configuration of an information processing system according to an embodiment. FIG. 4 is a diagram showing an example of the hardware configuration of an information processing device according to an embodiment. FIG. 5 is a diagram showing an example of moving image compression processing of an information processing device according to an embodiment. FIG. 6 is a diagram showing an example of motion vector compression processing of an information processing device according to an embodiment. FIG. 7 is a diagram showing an example of moving image restoration processing of an information processing device according to an embodiment.

[0013] The principles of the present disclosure will be described with reference to some exemplary embodiments. It should be understood that these embodiments are set forth for illustrative purposes only, to aid those skilled in the art in understanding and practicing the present disclosure, without implying any limitation on the scope of the disclosure. The disclosure described herein may be implemented in various ways other than those described below.

[0014] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0015] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each drawing is merely an example for describing one or more embodiments. Each drawing is not related to only one particular embodiment, but may also be related to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessarily required to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0016] (First Embodiment) <Configuration> <<Configuration of Information Processing Device 10>> The configuration of an information processing device 10 (encoder) that compresses moving images according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 10 that compresses moving images according to an embodiment. The information processing device 10 has a calculation unit 11 and a compression unit 12. Each of the calculation unit 11 and the compression unit 12 may be realized by cooperation between one or more programs installed in the information processing device 10 and hardware such as a processor and memory of the information processing device 10.

[0017] The calculation unit 11 calculates the amount of change of each pixel of the second frame relative to surrounding pixels based on a frame that is earlier than the first frame in the moving image. The compression unit 12 compresses the motion vector with a code amount corresponding to the amount of change at the position indicated by the motion vector corresponding to one or more pixels of the second frame, estimated based on the first and second frames.

[0018] <<Configuration of Information Processing Device 20>> Next, the configuration of the information processing device 20 (decoder) that restores a moving image according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 20 that restores a moving image according to the embodiment. The information processing device 20 has an acquisition unit 21 and a restoration unit 22. Each of the acquisition unit 21 and the restoration unit 22 may be realized by cooperation between one or more programs installed in the information processing device 20 and hardware such as a processor and memory of the information processing device 20.

[0019] The acquisition unit 21 acquires information indicating the amount of change of each pixel of the second frame relative to neighboring pixels based on a frame that is earlier than the first frame in the moving image, and motion vectors estimated based on the first and second frames and compressed with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame. The restoration unit 22 restores the compressed motion vectors acquired by the acquisition unit 21 based on the amount of change.

[0020] <System Configuration> Next, the configuration of an information processing system 1 according to an embodiment will be described with reference to FIG. 3 . FIG. 3 is a diagram illustrating an example of the configuration of the information processing system 1 according to an embodiment. In the example of FIG. 3 , the information processing system 1 includes an information processing device 10 and an information processing device 20. Note that the number of information processing devices 10 and information processing devices 20 is not limited to the example of FIG. 3 . Note that the technology of the present disclosure can be used in various services that encode and transmit video, such as, for example, distribution of stored video, compression of video for recording (saving) on ​​a recording medium, and distribution of video in real time. In this case, the technology of the present disclosure can also be used in remote medical treatment, for example, in which video captured by a camera (imaging device) such as an endoscope is distributed to a remote specialist and instructions are received from the specialist. Furthermore, the video of the present disclosure is not limited to video captured by an imaging device, and may be, for example, a video of distance images captured by a stereo camera or LiDAR (Light Detection and Ranging). Furthermore, the video of the present disclosure may be, for example, a video of thermal images captured by an infrared camera.

[0021] 3, the information processing device 10 and the information processing device 20 are connected to each other so as to be able to communicate with each other via a network N. Examples of the network N include the Internet, a mobile information processing system, a wireless LAN (Local Area Network), a short-range wireless communication such as BLE (Bluetooth Low Energy, registered trademark), a LAN, a bus, etc. Examples of the mobile information processing system include a fifth-generation mobile information processing system (5G, 5th Generation), a fourth-generation mobile information processing system (4G), a third-generation mobile information processing system (3G), etc.

[0022] Each of the information processing device 10 and the information processing device 20 may be, for example, a device such as a personal computer, a smartphone, a server, or a cloud server. The information processing device 10 compresses video. The information processing device 10 may also transmit the compressed video to the information processing device 20. The information processing device 20 expands (restores) the video compressed by the information processing device 10.

[0023] <Hardware Configuration> Fig. 4 is a diagram showing an example of the hardware configuration of the information processing device 10 and the information processing device 20 according to the embodiment. In the example of Fig. 4, each of the information processing device 10 and the information processing device 20 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These components may be connected via a bus or the like. The memory 102 stores at least a part of a program 104. The communication interface 103 includes an interface required for communication with other network elements.

[0024] When the program 104 is executed by the processor 101, memory 102, and other components in cooperation with each other, the computer 100 performs at least some of the processing of the embodiments of the present disclosure. The memory 102 may be of any type. As a non-limiting example, the memory 102 may be a non-transitory computer-readable storage medium. The memory 102 may also be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Although only one memory 102 is shown in the computer 100, several physically different memory modules may be present in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and, as a non-limiting example, a processor based on a multi-core processor architecture. The computer 100 may have multiple processors, such as application-specific integrated circuit chips that are time-slaved to a clock that synchronizes the main processor.

[0025] Embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device.

[0026] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, that execute on a target real or virtual processor or device to perform the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or divided among program modules as desired in various embodiments. The machine-executable instructions of the program modules may be executed in local or distributed devices. In a distributed device, the program modules may be located in both local and remote storage media.

[0027] The program code for executing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus. When the program code is executed by the processor or controller, the functions / acts in the flowcharts and / or implementing block diagrams are performed. The program code may be executed entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine, or entirely on a remote machine or server.

[0028] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0029] <Processing> <<Video Compression (Encoding) Processing>> Next, an example of video compression processing by the information processing device 10 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of video compression processing by the information processing device 10 according to the embodiment. Note that, hereinafter, the same components are assigned the same reference numerals to avoid redundant description.

[0030] In step S101, the compression unit 12 estimates a motion vector based on the frame to be compressed this time (first frame) 501 and the second frame 502, lossy compresses the estimated motion vector, and decompresses the compressed motion vector. This generates a restored (compressed and then decompressed) motion vector 503. The second frame 502 is, for example, a frame from a point in time before the first frame 501, and may be a compressed and restored frame. Details of the processing of step S101 in this case will be described later with reference to FIG. 6 .

[0031] The second frame 502 may be, for example, a frame that is earlier than the first frame 501 and is based on a compressed and restored frame. Details of the process of step S101 in this case will be described later with reference to FIG. 7.

[0032] Next, the compression unit 12 performs motion compensation based on the restored (expanded after compression) motion vector 503 and the second frame 502, and generates a motion-compensated frame 504 (step S102).

[0033] Next, the compression unit 12 generates a residual image 505 by calculating the difference (residual) for each pixel between the first frame 501 and the motion-compensated frame 504 (step S103). The compression unit 12 may determine the code rate of the motion vector and the code rate of the residual using, for example, rate-distortion optimization (RDO). In this case, the accuracy of the motion vector restored by the technology disclosed herein is improved, and the code rate of the residual required to achieve a specific image quality for the restored frame can also be reduced. Therefore, the total value of the code rate of the motion vector and the code rate of the residual can be further reduced, thereby improving the video compression efficiency.

[0034] Next, the compression unit 12 performs lossy compression on the residual image 505 and decompresses the compressed residual image (step S104). Next, the compression unit 12 adds, for each pixel, the residual image restored by the processing of step S104 and the motion-compensated frame 504 to generate a currently restored frame 506 (step S105). Note that the restored frame 506 is used as the previously restored frame 502 when compressing the next frame to be compressed.

[0035] <<<Motion Vector Compression Process (Part 1)>>> Next, an example of the motion vector compression process of the information processing device 10 according to the embodiment, which is performed in step S101 in Fig. 5, will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of the motion vector compression process of the information processing device 10 according to the embodiment.

[0036] In step S201, the compression unit 12 estimates a motion vector corresponding to one or more pixels of the second frame 502 based on the first frame 501 and the second frame 502. Here, the compression unit 12 may use, for example, learning results of deep learning to estimate (infer) a motion vector for each pixel or for each block including multiple pixels based on the first frame 501 and the second frame 502. This can improve, for example, the accuracy of estimating the motion vector.

[0037] Next, the calculation unit 11 calculates the amount of change for each of one or more pixels in the second frame 502 relative to its surrounding pixels (step S202). This generates a change amount MAP 602 indicating the amount of change for each of one or more pixels in the second frame 502 relative to its surrounding pixels. Here, the calculation unit 11 may use, for example, a convolutional neural network (CNN) to calculate the amount of change for each pixel or for each block including multiple pixels. The convolutional neural network may determine (estimate) a larger value for the amount of change based on at least one of the following conditions: the greater the color change of the subject relative to the surrounding pixels, the smaller the subject appears, the greater the unevenness of the subject, the more in-focus the subject is, and the less blur the subject experiences. In this case, for example, a relatively large value is determined for an area in which a person or the like relatively far from the camera is captured.

[0038] This allows the change amount to be set to a larger value for a location where a relatively small difference in position compared to surrounding pixels results in a relatively large residual. Note that the convolutional neural network may be generated by supervised learning using, for example, an image as an input (explanatory variable) and a change amount map in which the change amount is set to a larger value as the degree of 1 or higher described above increases as the degree increases.

[0039] The calculation unit 11 may calculate the amount of change for each pixel using, for example, a Laplacian filter, which is a filter that detects image edges using second-order derivatives. This allows the amount of change to be determined to be larger for areas with sharp edges, for example.

[0040] Next, the compression unit 12 lossy compresses the motion vector with a code amount corresponding to the amount of change at the position indicated by the estimated motion vector (step S203). Here, the compression unit 12 may, for example, determine a larger code amount the greater the amount of change at the position indicated by the estimated motion vector. This reduces the degree of compression of the motion vector at locations with greater change, thereby reducing degradation of the accuracy of the motion vector due to compression of the motion vector. In this case, the compression unit 12 may, for example, determine a smaller QP (Quantization Parameter) value or quantization width to be used for motion vector compression the greater the amount of change at the position indicated by the estimated motion vector. The position indicated by the motion vector may, for example, be the location of the referenced data in the second frame 502.

[0041] Furthermore, the direction of the motion vector may be calculated as the direction from a certain pixel in the first frame 501 toward the origin position of that pixel in the second frame 502. In this case, the position of the certain pixel in the second frame 502 is set as the end point of the motion vector, and the position to which that pixel has moved in the first frame 501 is set as the start point. In this case, the position indicated by the motion vector is the end point position of the motion vector. Note that the direction of movement of a certain pixel in the second frame 502 and the direction toward the origin position of a certain pixel in the first frame 501 are opposite directions. In this embodiment, the direction of the motion vector may be defined or calculated by either method.

[0042] The compression unit 12 may compress the motion vector with a code amount corresponding to the amount of change for the pixel at the position indicated by the motion vector. Alternatively, the compression unit 12 may compress the motion vector with a code amount corresponding to the amount of change for each of a plurality of pixels surrounding the position indicated by the motion vector. In this case, the compression unit 12 may determine the code amount by, for example, taking a weighted average of the amount of change for the pixel at the position indicated by the motion vector and the amount of change for a specific number of pixels surrounding the position indicated by the motion vector, with the weighting being larger the closer the pixel is to the position indicated by the motion vector.

[0043] Next, the compression unit 12 expands the compressed motion vector 603 (step S204), thereby generating the restored motion vector 503.

[0044] <<<<Motion Vector Compression Process (Part 2)>>> Fig. 6 has described an example in which a frame that is an earlier frame than the first frame 501 and that has been compressed and restored is used as the second frame 502. Fig. 7 has described an example in which a frame that is an earlier frame than the first frame 501 and that is based on a frame that has been compressed and restored is used as the second frame 502. In the example of Fig. 7, compared to the example of Fig. 6, for example, the processing load increases, but compression efficiency can be further improved.

[0045] Another example of the motion vector compression process of the information processing device 10 according to the embodiment in step S101 of Fig. 5 will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the motion vector compression process of the information processing device 10 according to the embodiment.

[0046] In step S301, the compression unit 12 predicts a motion vector for a previously restored frame and the first frame 501, which is the frame to be compressed this time, to generate a predicted motion vector 702. Here, the compression unit 12 may generate the predicted motion vector 702 based on data 701 such as a plurality of previously restored frames and a motion vector based on the plurality of frames and previously restored.

[0047] Next, the compression unit 12 performs motion compensation based on the previously restored frame 703 and the predicted motion vector 702 to generate the second frame 502, which is a predicted frame for the first frame 501 (step S302).

[0048] Next, the compression unit 12 estimates a motion vector corresponding to one or more pixels of the second frame based on the first frame 501 and the second frame 502 (step S303). Next, the calculation unit 11 calculates the amount of change of each of one or more pixels of the second frame 502 relative to its surrounding pixels (step S304). Next, the compression unit 12 lossy-compresses the motion vector with a code amount corresponding to the amount of change at the position indicated by the estimated motion vector (step S305). Next, the compression unit 12 decompresses the compressed motion vector 705 (step S306). Note that the processes of steps S303 to S306 may be the same as the processes of steps S201 to S204 in FIG. 6 . Note that the process of step S306 generates a motion vector 706 based on the first frame 501, which is the frame to be compressed this time, and the second frame 502, which is the predicted frame.

[0049] Next, the compression unit 12 performs motion compensation processing based on the predicted motion vector 702 and the motion vector 706 (step S307). Next, the compression unit 12 adds the motion vector 706 to the compensation processing result (step S308). As a result, the restored motion vector 503 is generated.

[0050] <<Video Restoration (Decoding) Process>> Next, an example of video restoration process of the information processing device 20 according to the embodiment will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of video restoration process of the information processing device 20 according to the embodiment.

[0051] In step S401, the acquisition unit 21 acquires a motion vector compressed with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels in the second frame 502, and a compressed residual image 505A. Note that the motion vector compressed with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels in the second frame 502 is the motion vector 603 in FIG. 6 or the motion vector 705 in FIG. 7. Note that the second frame 502 may be, for example, a frame that is earlier than the first frame 501 described in FIG. 6 and that has been compressed and restored. Also, the second frame 502 may be, for example, a frame that is earlier than the first frame 501 described in FIG. 7 and that is based on a frame that has been compressed and restored.

[0052] Next, the restoration unit 22 expands the compressed motion vector acquired by the acquisition unit 21 based on the amount of change of each pixel in the previously restored second frame 502 relative to its surrounding pixels (step S402). Note that the processing of step S402 may be the same as the processing of step S204 in Fig. 6 or the processing of step S306 in Fig. 7. In this way, a restored motion vector 503 is generated. Note that information indicating the amount of change of each pixel in the previously restored second frame 502 relative to its surrounding pixels is, for example, the change amount MAP 602 in Fig. 6 or 7.

[0053] Next, the restoration unit 22 performs motion compensation based on the restored motion vector 503 and the second frame 502, and generates a motion-compensated frame 504 (step S403).

[0054] Next, the restoration unit 22 decompresses the compressed residual image 505A (step S404). Next, the restoration unit 22 adds the restored residual image 505 and the motion-compensated frame 504 for each pixel to generate a currently restored frame 506 (step S405). Note that the restored frame 506 is used as the previously restored frame 502 when restoring the next frame to be restored.

[0055] <Others> It is conceivable to use deep learning, which has been a subject of research in recent years, to estimate (infer) a motion vector for each pixel or for each block containing multiple pixels, based on the frame 501 to be compressed this time and a previously restored frame 502. In this case, the estimation accuracy is improved compared to conventional methods, but since the motion vector is estimated for each pixel or each relatively small block, the data size increases due to the high resolution of the motion vector.

[0056] Compressed video data includes compressed data of motion vectors for each frame. Therefore, if the data size of the motion vectors increases, the data size of the compressed video will also increase if the compression rate of the motion vectors is the same as before. Furthermore, if the compression rate of the motion vectors is increased, the accuracy of the restored motion vectors will deteriorate.

[0057] On the other hand, the technology disclosed herein allows for appropriate compression of motion vectors. As a result, for example, it is possible to improve the compression rate of moving images while maintaining the quality of the moving images. It is also possible to improve the quality of moving images while maintaining the same compression rate as conventional techniques.

[0058] <Modifications> The information processing device 10 and the information processing device 20 may each be a device included in a single housing, but the information processing device 10 and the information processing device 20 of the present disclosure are not limited to this. Each unit of the information processing device 10 and the information processing device 20 may be realized, for example, by cloud computing configured with one or more computers. Furthermore, at least a portion of the processing of the information processing device 10 may be executed, for example, by the information processing device 20. Furthermore, at least a portion of the processing of the information processing device 20 may be executed, for example, by the information processing device 10. Furthermore, the information processing device 10 and the information processing device 20 may be an integrated device. Such information processing devices 10 and information processing devices 20 are also included in examples of the "information processing device" of the present disclosure.

[0059] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0060] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes. Note that some or all of the elements (e.g., configurations and functions) described in each supplementary note dependent on supplementary note 1 may also be dependent on independent supplementary notes in other categories in a similar dependency relationship. Some or all of the elements described in any supplementary note may be applied to various hardware, software, and recording means, systems, and methods for recording software. (Supplementary note 1) An information processing device comprising: a calculation unit that calculates an amount of change for each pixel of a second frame based on a frame that is earlier than a first frame in a moving image with respect to neighboring pixels; and a compression unit that compresses a motion vector with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame. (Supplementary note 2) The information processing device described in supplementary note 1, wherein the compression unit compresses the motion vector with a code amount corresponding to at least one of the amount of change for the pixel at the position indicated by the motion vector and the amount of change for each of a plurality of pixels surrounding the position indicated by the motion vector. (Supplementary Note 3) The information processing device according to Supplementary Note 1 or 2, wherein the calculation unit calculates the amount of change for each pixel using a convolutional neural network that determines the value of the amount of change to be larger in accordance with at least one of the following conditions: the smaller the subject appears, the more in focus the subject, and the less blur there is of the subject. (Supplementary Note 4) The information processing device according to Supplementary Note 1 or 2, wherein the calculation unit calculates the amount of change for each pixel using a convolutional neural network that determines the value of the amount of change to be larger in accordance with at least one of the following conditions: the greater the change in color of the subject relative to surrounding pixels and the greater the unevenness of the subject. (Supplementary Note 5) The information processing device according to Supplementary Note 1 or 2, wherein the calculation unit calculates the amount of change for each pixel using a Laplacian filter. (Supplementary Note 6) The information processing device according to Supplementary Note 1 or 2, wherein the compression unit determines the code rate of a motion vector and the code rate of a residual using rate-distortion optimization (RDO).(Supplementary Note 7) An information processing method comprising: calculating an amount of change of each pixel of a second frame based on a frame that is earlier than a first frame in a moving image relative to neighboring pixels; and compressing the motion vector with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame. (Supplementary Note 8) An information processing device comprising: an acquisition unit that acquires the motion vector, which is estimated based on a first frame in a moving image and a second frame based on a frame that is earlier than the first frame, and compressed with a code amount corresponding to the amount of change of each pixel of the second frame relative to neighboring pixels at a position indicated by the motion vector corresponding to one or more pixels of the second frame; and a restoration unit that restores the compressed motion vector acquired by the acquisition unit based on the amount of change. (Supplementary Note 9) An information processing method comprising: obtaining a motion vector estimated based on a first frame of a moving image and a second frame based on a frame earlier than the first frame, the motion vector being compressed with a code amount corresponding to an amount of change in each pixel of the second frame relative to surrounding pixels at a position indicated by the motion vector corresponding to one or more pixels of the second frame; and restoring the compressed motion vector based on the amount of change. (Supplementary Note 10) An information processing system comprising a first information processing device and a second information processing device, wherein the first information processing device comprises: a calculation unit that calculates an amount of change for each pixel of a second frame based on a frame that is earlier than a first frame in a moving image with respect to surrounding pixels; and a compression unit that compresses the motion vector with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame; and the second information processing device comprises: an acquisition unit that acquires the motion vector compressed with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame; and a restoration unit that restores the compressed motion vector acquired by the acquisition unit based on the amount of change.(Supplementary Note 11) A program that causes a computer to execute the following process: calculate an amount of change for each pixel of a second frame based on a frame that is earlier than a first frame in a moving image, and compress the motion vector with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame. (Supplementary Note 12) A program that causes a computer to execute the following process: acquire the motion vector that is estimated based on a first frame in a moving image and a second frame based on a frame that is earlier than the first frame, and compressed with a code amount corresponding to the amount of change for each pixel of the second frame, at a position indicated by a motion vector corresponding to one or more pixels of the second frame, and restore the compressed motion vector based on the amount of change.

[0061] This application claims priority based on Japanese Patent Application No. 2024-110850, filed July 10, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0062] REFERENCE SIGNS LIST 1 Information processing system 10 Information processing device 11 Calculation unit 12 Compression unit 20 Information processing device 21 Acquisition unit 22 Restoration unit

Claims

1. An information processing device having: a calculation unit that calculates the amount of change in each pixel of a second frame based on a frame that is earlier than a first frame in a moving image, relative to surrounding pixels; and a compression unit that compresses the motion vector with a code amount that corresponds to the amount of change at a position indicated by the motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame.

2. The information processing device according to claim 1, wherein the compression unit compresses the motion vector with a code amount corresponding to at least one of the amount of change for the pixel at the position indicated by the motion vector and the amount of change for each of a plurality of pixels surrounding the position indicated by the motion vector.

3. The information processing device according to claim 1 or 2, wherein the calculation unit calculates the amount of change for each pixel using a convolutional neural network that determines the value of the amount of change to be a larger value in accordance with at least one of the following conditions: the smaller the subject is photographed, the more in focus the subject is, and the less blur there is in the subject.

4. The information processing device according to claim 1 or 2, wherein the calculation unit calculates the amount of change for each pixel using a convolutional neural network that determines the value of the amount of change to be a larger value in accordance with at least one of the greater the change in color of the subject relative to surrounding pixels and the greater the unevenness of the subject.

5. The information processing device according to claim 1 or 2, wherein the calculation unit calculates the amount of change for each pixel using a Laplacian filter.

6. The information processing device according to claim 1 or 2, wherein the compression unit determines the code amount of the motion vector and the code amount of the residual using rate-distortion optimization (RDO).

7. An information processing method comprising: calculating the amount of change in each pixel of a second frame based on a frame that is earlier than a first frame in a moving image relative to surrounding pixels; and compressing the motion vector with a code amount corresponding to the amount of change at a position indicated by the motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame.

8. The information processing method according to claim 7, wherein the motion vector is compressed with a code amount corresponding to at least one of the amount of change for the pixel at the position indicated by the motion vector and the amount of change for each of a plurality of pixels surrounding the position indicated by the motion vector.

9. The information processing method according to claim 7 or 8, wherein the amount of change for each pixel is calculated using a convolutional neural network that determines the amount of change to be a larger value in accordance with at least one of the following conditions: the smaller the subject is photographed, the more in focus the subject is, and the less blur there is in the subject.

10. An information processing method according to claim 7 or 8, wherein the amount of change for each pixel is calculated using a convolutional neural network that determines the amount of change to be a larger value in accordance with at least one of the following: the greater the change in color of the subject relative to surrounding pixels, and the greater the unevenness of the subject.

11. The information processing method according to claim 7 or 8, wherein the amount of change for each pixel is calculated using a Laplacian filter.

12. The information processing method according to claim 7 or 8, wherein the code rate of the motion vector and the code rate of the residual are determined using rate-distortion optimization (RDO).

13. An information processing device having: an acquisition unit that acquires a motion vector estimated based on a first frame of a video and a second frame based on a frame earlier than the first frame, the motion vector being compressed with a code amount corresponding to an amount of change in each pixel of the second frame relative to surrounding pixels at a position indicated by the motion vector corresponding to one or more pixels of the second frame; and a restoration unit that restores the compressed motion vector acquired by the acquisition unit based on the amount of change.

14. An information processing method comprising: obtaining a motion vector estimated based on a first frame of a video and a second frame based on a frame earlier than the first frame, the motion vector corresponding to one or more pixels of the second frame, and compressed with a code amount corresponding to the amount of change of each pixel of the second frame relative to surrounding pixels at a position indicated by the motion vector; and restoring the compressed motion vector based on the amount of change.

15. An information processing system comprising a first information processing device and a second information processing device, wherein the first information processing device comprises: a calculation unit that calculates an amount of change for each pixel of a second frame based on a frame that is earlier than the first frame in a moving image with respect to surrounding pixels; and a compression unit that compresses a motion vector with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame, estimated based on the first frame and the second frame; and the second information processing device comprises: an acquisition unit that acquires the motion vector compressed with a code amount corresponding to the amount of change at a position indicated by a motion vector corresponding to one or more pixels of the second frame; and a restoration unit that restores the compressed motion vector acquired by the acquisition unit based on the amount of change.

16. The information processing system according to claim 15, wherein the compression unit compresses the motion vector with a code amount corresponding to at least one of the amount of change for the pixel at the position indicated by the motion vector and the amount of change for each of a plurality of pixels surrounding the position indicated by the motion vector.

17. An information processing system as described in claim 15 or 16, wherein the calculation unit calculates the amount of change for each pixel using a convolutional neural network that determines the value of the amount of change to be a larger value in accordance with at least one of the following conditions: the smaller the subject is photographed, the more in focus the subject is, and the less blur there is in the subject.

18. An information processing system as described in claim 15 or 16, wherein the calculation unit calculates the amount of change for each pixel using a convolutional neural network that determines the value of the amount of change to be larger depending on at least one of the following: the greater the change in color of the subject relative to surrounding pixels and the greater the unevenness of the subject.

19. A program that causes a computer to execute the following process: calculate the amount of change in each pixel of a second frame based on a frame that is earlier than a first frame in a video, relative to its surrounding pixels; and compress the motion vector with a code amount corresponding to the amount of change at the position indicated by the motion vector, which is estimated based on the first frame and the second frame, and corresponds to one or more pixels of the second frame.

20. A program that causes a computer to execute a process of obtaining a motion vector that is estimated based on a first frame of a video and a second frame that is based on a frame that occurs earlier than the first frame, and that is compressed with a code amount that corresponds to the amount of change in each pixel of the second frame relative to surrounding pixels at the position indicated by the motion vector corresponding to one or more pixels of the second frame, and restoring the compressed motion vector based on the amount of change.

Images