Information processing system, information processing method, and information processing apparatus

Abstract

This information processing system (1) comprises: an acquisition means (11) that, for the frames of a video delivered via a network, acquires information indicating the vectors of movements related to the respective ones of small areas obtained by dividing each frame of the video into a plurality of areas; an estimation means (12) that estimates the position of the area of a particular part of a subject in the frames of the video on the basis of the information acquired by the acquisition means and indicating the vectors of the movements; and a control means (13) that, on the basis of the position of the area of the particular part estimated by the estimation means, causes encoding parameters to be specified with respect to the frames of the video.

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing apparatus.

Background Art

[0002] There is an increasing interest in a technique of collecting information detected by a sensor via a network and determining the situation of a target based on the collected information.

[0003] In relation to this technique, Patent Document 1 describes that after taking a patient's affected part image (still image) using a medical imaging device, the data of the taken affected part image is transferred to a portable information terminal possessed by a specialist, so that a specialist located remotely can grasp the situation of the affected part.

[0004] Also, Patent Document 2 discloses a technique for enabling examination without losing sight of an affected part in a telemedicine system in which a doctor and a patient can have a consultation while conversing, by moving a selection area set in the patient's video in accordance with the patient's movement. In Patent Document 2, in order to extract movement information from video data, for each fixed time, the change in pixels at the same position is compared between the previous screen and the current screen. Then, correlation calculation is performed on the change in pixels at the same position across the entire screen, and it is disclosed that the change in the movement of the screen is quantitatively measured based on how much the correlation value changes between the previous screen and the current screen.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, Patent Document 1 does not discuss how to handle the case where analysis is performed based on video. In addition, Patent Document 2 has a problem that when the image quality of the affected area in the video is not sufficient, analysis cannot be performed with a certain accuracy.

[0007] An object of the present disclosure is to provide a technique capable of appropriately distributing a video of an area of a specific part of a subject used for inspection (analysis, estimation, inference, diagnosis) in view of the above-described problems.

Means for Solving the Problems

[0008] In a first aspect according to the present disclosure, an information processing system includes: an acquisition unit that acquires information indicating a movement vector for each of small regions obtained by dividing a frame of a video distributed via a network into a plurality of regions; an estimation unit that estimates a position of an area of a specific part of a subject in the frame of the video based on the information indicating the movement vector acquired by the acquisition unit; and a control unit that causes encoding parameters to be set for the frame of the video based on the position of the area of the specific part estimated by the estimation unit.

[0009] Also, in a second aspect according to the present disclosure, there is provided an information processing method for executing, for a frame of a video distributed via a network, a process of acquiring information indicating a movement vector for each of small regions obtained by dividing the frame of the video into a plurality of regions, a process of estimating a position of an area of a specific part of a subject in the frame of the video based on the information indicating the movement vector acquired in the acquiring process, and a process of causing encoding parameters to be set for the frame of the video based on the position of the area of the specific part estimated in the estimating process.

[0010] Also, in a third aspect according to the present disclosure, an information processing apparatus includes: acquisition means for acquiring, for each small region obtained by dividing a frame of a video distributed via a network into a plurality of regions, information indicating a movement vector regarding each small region; estimation means for estimating a position of a region of a specific part of a subject in the frame of the video based on the information indicating the movement vector acquired by the acquisition means; and control means for setting encoding parameters for the frame of the video based on the position of the region of the specific part estimated by the estimation means.

Advantages of the Invention

[0011] According to one aspect, it is possible to appropriately distribute a video of a region of a specific part of a subject used for inspection.

Brief Description of the Drawings

[0012]

Figure 1A

Figure 1B

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0013] The principles of the present disclosure are described with reference to several exemplary embodiments. These embodiments are described for illustrative purposes only and are not intended to suggest limitations on the scope of the present disclosure, and it should be understood that they are intended to assist those skilled in the art in understanding and implementing the present disclosure. The disclosure described herein may be implemented in various ways other than those described below. In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which the present disclosure belongs. Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0014] <First Embodiment> <Configuration> Referring to FIG. 1A, the configuration of the information processing system 1 according to the embodiment will be described. FIG. 1A is a diagram showing an example of the configuration of the information processing system 1 according to the embodiment. The information processing system 1 includes an acquisition unit 11, an estimation unit 12, and a control unit 13.

[0015] The acquisition unit 11 receives (acquires) various types of information from the storage unit inside the device or an external device. Further, the acquisition unit 11 may receive an image from the imaging device 20 built in the device via an internal bus. Also, the acquisition unit 11 may receive an image from an external (external attachment) imaging device 20 connected by a cable or the like via an external bus (for example, a USB (Universal Serial Bus) cable, an HDMI (registered trademark) (High-Definition Multimedia Interface) cable, an SDI cable). In this case, the external bus may include, for example, a USB (Universal Serial Bus) cable, an HDMI (High-Definition Multimedia Interface) cable, or an SDI (Serial Digital Interface) cable or the like.

[0016] Also, the acquisition unit 11 acquires, for example, information indicating a motion vector for each small area obtained by dividing the frame of the video distributed via the network N into a plurality of areas, regarding the frame of the video. The estimation unit 12 estimates the position of the area of the specific part of the subject in the frame based on the information indicating the motion vector acquired by the acquisition unit 11. The control unit 13 executes various processes based on the image captured and distributed by the imaging device 20. The control unit 13 causes an encoding parameter based on the position of the area of the specific part estimated by the estimation unit 12 to be set for the frame. Note that the encoding parameter based on the position of the area of the specific part may include at least one of, for example, the encoding bit rate, the encoding frame rate, and the encoding quantization parameter (QP value) for each small area included in the area of the specific part.

[0017] Further, the acquisition unit 11, the estimation unit 12, and the control unit 13 may be integrated into one device as shown in FIG. 1B. In the example of FIG. 1B, the information processing system 1 includes an information processing device 10 and a photographing device 20. The photographing device 20 is a device that photographs a subject, and may be, for example, a camera built into a smartphone, a tablet, or the like. Further, the photographing device 20 may be, for example, a camera connected to a personal computer or the like via an external bus. The information processing device 10 includes an acquisition unit 11, an estimation unit 12, and a control unit 13. These units may be realized by the cooperation of one or more programs installed in the information processing device 10, the processor 101 of the information processing device 10, and hardware such as the memory 102.

[0018] <Processing> Next, with reference to FIGS. 2A and 2B, an example of the processing of the information processing system 1 according to the embodiment will be described. FIG. 2A is a flowchart showing an example of the processing of the information processing system 1 according to the embodiment. FIG. 2B is a diagram showing an example of each small area and motion vector in the frame according to the embodiment.

[0019] In step S1, the acquisition unit 11 acquires, for example, information indicating the movement vector of each of the small areas obtained by dividing the frame into a plurality of areas for a frame of video encoded by a predetermined encoding method. The encoding method may include, for example, H.265 / HEVC (High Efficiency Video Coding), AV1 (AOMedia Video 1), H.264 / MPEG-4 AVC (Advanced Video Coding), or the like. Further, the small area may be, for example, an encoding macroblock or an encoding PU (Predicted Unit). Further, the information indicating the movement vector may be, for example, a motion vector (MV: Motion Vector) used in motion compensation (MC: Motion Compensation) in inter-frame prediction of encoding.

[0020] Subsequently, the estimation unit 12 estimates the position of the region of the specific part of the subject in the frame based on the information indicating the movement vector acquired by the acquisition unit 11 (step S2). FIG. 2B illustrates examples of each small region 203A to D included in the region 202 of the specific part and the motion vectors 204A to D of each small region 203A to D in the frame 201 included in the video. For example, the estimation unit 12 may estimate, as the region of the small region 203A in the next frame of the frame 201, the region on the pixel coordinates obtained by moving the small region 203A in the direction and by the amount of movement indicated by the motion vector 204A. Similarly, the estimation unit 12 may estimate, as the regions of the small regions 203B to D in the next frame, the regions on the pixel coordinates obtained by moving each of the small regions 203B to D in the direction and by the amount of movement indicated by each of the motion vectors 204B to D. Then, for example, the estimation unit 12 may estimate, as the region of the specific part in the next frame, the region including the regions of the small regions 203A to D in the next frame.

[0021] Subsequently, the control unit 13 causes the encoding parameters based on the estimated position of the region of the specific part to be set for the frame (step S3). Here, for example, the control unit 13 causes the region of the specific part in the next frame of the frame 201 to be encoded with a specific image quality. Thereby, for example, the region of the specific part (region of interest) used for analysis can be encoded and distributed with a higher image quality (for example, bit rate, frame rate, QP value) than other regions (regions other than the specific part).

[0022] <Hardware Configuration> FIG. 3 is a diagram showing a hardware configuration example of the information processing apparatus 10 according to the embodiment. In the example of FIG. 3, the information processing apparatus 10 (computer 100) includes a processor 101, a memory 102, and a communication interface 103. These components may be connected by a bus or the like. The memory 102 stores at least a part of the program 104. The communication interface 103 includes an interface necessary for communication with other network elements.

[0023] When program 104 is executed through the cooperation of the processor 101, the memory 102, etc., at least part of the processing of the embodiments of the present disclosure is performed by the computer 100. The memory 102 may be of any type suitable for a local technical network. The memory 102 may be, as a non-limiting example, a non-transitory computer-readable storage medium. Also, the memory 102 may be implemented using any suitable data storage technology such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. Although only one memory 102 is shown for the computer 100, there may be several physically different memory modules in the computer 100. The processor 101 may be of any type. The processor 101 may include one or more of a general-purpose computer, a dedicated computer, a microprocessor, a digital signal processor (DSP), and, as a non-limiting example, a processor based on a multi-core processor architecture. The computer 100 may have multiple processors such as an application-specific integrated circuit chip that is temporally dependent on a clock that synchronizes the main processor.

[0024] The embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that can be executed by a controller, a microprocessor, or other computing device.

[0025] The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions such as instructions included in program modules, which are executed on a device on a target physical processor or virtual processor to execute the processes or methods of the present disclosure. Program modules include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The functions of the program modules may be combined or divided among the program modules as desired in various embodiments. The machine-executable instructions of the program modules can be executed within a local or distributed device. In a distributed device, the program modules can be arranged on both local and remote storage media.

[0026] The program code for executing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programmable data processing devices. When the program code is executed by the processor or controller, the functions / operations in the flowchart and / or the implementation block diagram are executed. The program code is executed entirely on the machine, partly on the machine as a stand-alone software package, partly on the machine, partly on a remote machine, or entirely on a remote machine or server.

[0027] The program can be stored using various types of non-transitory computer-readable media and supplied to a computer. Non-transitory computer-readable media include various types of tangible recording media. Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, optical disk media, semiconductor memories, etc. Magnetic recording media include, for example, flexible disks, magnetic tapes, hard disk drives, etc. Magneto-optical recording media include, for example, magneto-optical disks, etc. Optical disk media include, for example, Blu-ray disks, CD (Compact Disc)-ROM (Read Only Memory), CD-R (Recordable), CD-RW (ReWritable), etc. Semiconductor memories include, for example, solid state drives, mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAM (random access memory), etc. Also, the program may be supplied to the computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or wireless communication channels.

[0028] <Second Embodiment> <System Configuration> Next, with reference to FIG. 4, the configuration of the information processing system 1 according to the embodiment will be described. FIG. 4 is a diagram showing a configuration example of the information processing system 1 according to the embodiment. In the example of FIG. 4, the information processing system 1 includes an information processing device 10 having a photographing device 20 and a distribution destination device 30. Note that the number of the information processing device 10 and the distribution destination device 30 is not limited to the example of FIG. 4.

[0029] Delivery destination device 30 Note that the technology of the present disclosure may be used, for example, in the measurement of biological information based on an image of a patient in a video conference (video call, online medical examination) between a doctor and a patient (human, animal). Further, the technology of the present disclosure may be used, for example, in the analysis (identification) of a person and the analysis (estimation) of behavior based on an image of a surveillance camera. Further, the technology of the present disclosure may be used, for example, in the analysis (inspection) of a product based on an image of a surveillance camera in a factory or a plant.

[0030] In the example of FIG. 4, the information processing device 10 and the delivery destination device 30 are connected so as to be able to communicate via the network N. Examples of the network N include, for example, the Internet, a mobile communication system, a wireless LAN (Local Area Network), Wi-Fi (registered trademark), a LAN, and short-range wireless communication such as BLE (Bluetooth (registered trademark) Low Energy). Examples of the mobile communication system include, for example, the fifth-generation mobile communication system (5G), local 5G, Beyond 5G (6G), the fourth-generation mobile communication system (4G), LTE (Long Term Evolution), and the third-generation mobile communication system (3G).

[0031] The information processing device 10 may be, for example, a device such as a smartphone, a tablet, or a personal computer. The information processing device 10 encodes an image (including a still image and a moving image (video)) captured by a built-in or external imaging device (camera) 20 by an arbitrary encoding method and distributes it to the delivery destination device 30 via the network N. The encoding method may include, for example, H.265 / HEVC (High Efficiency Video Coding), AV1 (AOMedia Video 1), H.264 / MPEG-4 AVC (Advanced Video Coding), and the like.

[0032] The destination device 30 may be, for example, a device such as a personal computer, a server, the cloud, a smartphone, or a tablet. The destination device 30 may perform analysis based on the image distributed from the information processing device 10. Alternatively, the destination device 30 may decode the distributed video and display it on a display device. Thereby, a doctor or the like can visually analyze the patient's information remotely.

[0033] <Processing> Next, with reference to FIGS. 5 to 10, an example of the processing of the information processing system 1 according to the embodiment will be described. FIG. 5 is a sequence diagram showing an example of the processing of the information processing system 1 according to the embodiment. FIG. 6 is a diagram showing an example of the specific part DB (database) 601 according to the embodiment. FIG. 7 is a diagram showing an example of the motion vector of each small area in the first frame according to the embodiment. FIG. 8 is a diagram showing an example of the estimated position of each small area of the first frame in the second frame according to the embodiment. FIG. 9 is a diagram showing an example of the motion vector of each small area in the first frame according to the embodiment. FIG. 10 is a diagram showing an example of the distributed video according to the embodiment.

[0034] Hereinafter, as an example, the case of performing medical treatment or biometric information measurement based on a patient's image in a video conference (video call, online medical treatment) between a doctor and a patient will be described. Hereinafter, it is assumed that the processing such as the establishment of a video conference session has already been completed between the patient information processing device 10 and the doctor's destination device 30.

[0035] In step S101, the control unit 13 of the information processing device 10 causes the first frame of the video captured by the imaging device 20 to be encoded. Here, the control unit 13 of the information processing device 10 may encode, for example, the area of a specific part (for example, eyes, mouth, cheeks, etc.) of the subject in the first frame with a specific image quality. Further, the control unit 13 of the information processing device 10 may encode, for example, the area other than the area of the specific part of the subject in the first frame with an image quality lower than the specific image quality. Thereby, the usage bandwidth of the network N due to the distribution of the video can be reduced.

[0036] Note that the specific part may be designated by a doctor, for example. In this case, for example, the destination device 30 may be designated by an operation of surrounding a specific part of the patient while dragging it with a mouse or the like on the display screen of the patient's video. Also, the destination device 30 may be designated (selected) by the doctor from a list of specific parts of the subject. Further, the destination device 30 may be designated (selected) by the doctor from a list of items of biological information that are items to be analyzed (hereinafter also appropriately referred to as "analysis targets").

[0037] Then, the information processing device 10 may receive information indicating the specific part designated by the doctor from the destination device 30. When an item of biological information to be analyzed is designated, the estimation unit 12 of the information processing device 10 may refer to, for example, the specific part DB 601 and extract information on the specific part corresponding to the analysis target. Note that the specific part DB 601 may be stored (registered, set) in the storage device inside the information processing device 10, or may be stored in a DB server or the like outside the information processing device 10. In the example of FIG. 6, in the image quality change content DB 601, the specific part of the subject is recorded in association with the item of biological information to be analyzed. In the example of FIG. 6, for example, when analyzing the heart rate, it is defined that the area of the face is used and the like.

[0038] The estimation unit 12 of the information processing device 10 may perform image recognition on the first frame and detect (estimate) an area including the specific part of the subject designated by a doctor or the like. Note that the estimation unit 12 of the information processing device 10 may execute the process of performing image recognition on the area of the specific part at a predetermined time interval (for example, every 1 second), for example. Also, the estimation unit 12 of the information processing device 10 may use an I frame (Intra-coded Frame, intra-frame, key frame) that is encoded without using inter-frame prediction as the first frame.

[0039] Then, the control unit 13 of the information processing apparatus 10 causes the area of the specific part in the first frame to be encoded with a specific image quality. Thereby, the area used for inspection or the like can be made to have a high image quality. Here, the control unit 13 of the information processing apparatus 10 may cause the area of the specific part in the first frame to be encoded with a specific image quality by setting at least one of the encoding bit rate, the encoding frame rate, and the encoding quantization parameter (QP (Quantization Parameter) value) to a specific value. In this case, the control unit 13 of the information processing apparatus 10 may encode the first frame using, for example, a map (QP map) that sets the quantization parameter (QP value) of encoding for each specific pixel area unit (for example, 16 pixels in vertical direction × 16 pixels in horizontal direction). Further, when hierarchical encoding (SVC, Scalable Video Coding) is used as the encoding method, the control unit 13 of the information processing apparatus 10 may use the entire first frame as the base layer and the area of the specific part as the enhancement layer.

[0040] Subsequently, the control unit 13 of the information processing apparatus 10 causes the encoded first frame to be distributed (transmitted) to the destination apparatus 30 via the network N (step S102). Subsequently, the acquisition unit 11 of the information processing apparatus 10 acquires information indicating the motion vectors of each small area (for example, macroblock or PU (Predicted Unit)) that divides the frame into a plurality of areas (step S103). Here, the acquisition unit 11 of the information processing apparatus 10 may acquire, for example, information indicating the motion vectors of each small area that divides the video frame into a plurality of areas from a module or the like that performs encoding processing inside the information processing apparatus 10. Also, information indicating the motion vectors may be acquired by analyzing the data output as a result of the encoding processing.

[0041] The acquisition unit 11 of the information processing apparatus 10 may acquire, for example, a motion vector (MV) used in motion compensation (MC) in inter-frame prediction. Note that inter-frame prediction is, for example, a method of predicting a frame at a certain point in time based on one or more frames captured at different points in time (timings), and encoding the difference between the image of the predicted frame and the image of the frame captured at that point in time.

[0042] Subsequently, the estimation unit 12 of the information processing apparatus 10 estimates the position of the region of the specific part of the subject in the second frame based on the information indicating the vector of the movement acquired by the acquisition unit 11 (step S104). Note that the second frame is a frame captured by the imaging apparatus 20 at a point in time different from that of the first frame. The second frame may be, for example, a frame (P frame (Predicted Frame)) encoded using only forward prediction in inter-frame prediction. Alternatively, it may be a frame (B frame (Bi-directional Predicted Frame)) in which any one of forward prediction, backward prediction, and bidirectional prediction is selected and encoded.

[0043] According to an embodiment of the present disclosure, it is possible to track the position of a region (region of interest) of a specific part that is encoded with high image quality by using information on small regions and motion vectors used for encoding video. Therefore, for example, compared with the case of recognizing an object in the region of a specific part in each frame, the processing amount and power consumption can be reduced, and the processing can be speeded up. When recognizing an object in the region of a specific part in each frame, if the information processing apparatus 10 does not have hardware such as a GPU (Graphics Processing Unit) for performing object recognition, the CPU (Central Processing Unit) and software execute the object recognition process, so the time and power consumption required for the object recognition process increase. On the other hand, in the present disclosure, the specific part is tracked by using the information calculated during encoding. Therefore, when the information processing apparatus 10 has a circuit for encoding video, such as a smartphone, the tracking process of the present disclosure can be executed more quickly and with lower power consumption.

[0044] The estimation unit 12 of the information processing apparatus 10 may estimate the position of the region of the specific part in the second frame, for example, based on the motion vector calculated when predicting from the first frame to the second frame. In this case, the estimation unit 12 of the information processing apparatus 10 may calculate, for example, the positions obtained by moving from the positions of the respective small regions included in the region of the specific part in the first frame in the directions and amounts of movement indicated by the motion vectors of the respective small regions. Then, the estimation unit 12 of the information processing apparatus 10 may estimate, for example, the calculated respective positions as the positions of the respective small regions in the second frame. Here, when the first frame is a frame encoded without using inter-frame prediction, the estimation unit 12 of the information processing apparatus 10 may estimate the position of the region of the specific part in the second frame based on, for example, the motion vector of the frame encoded immediately before the first frame.

[0045] FIG. 7 illustrates examples of motion vectors 704A to 704D of each small region 703A to 703D included in the cheek region 702 in a part 701 of the first frame. FIG. 8 illustrates examples of regions 803A to 803D obtained by moving each small region 703A to 703D by each motion vector 704A to 704D in a part 801 of the second frame. The estimation unit 12 of the information processing apparatus 10 may estimate, for example, that a region 803A on the pixel coordinates obtained by moving the small region 703A in the direction and by the amount of movement indicated by the motion vector 704A is the region of the small region 703A in the second frame in the first frame.

[0046] (Example of estimating the position of the destination of a specific part based on characteristic parts) The estimation unit 12 of the information processing apparatus 10 may estimate the position of the region of a specific part in the second frame based on the motion vectors of each small region included in the region of a predetermined part (for example, eyes, nose, mouth, etc.) of the subject in the first frame. Thereby, for example, even when the specific part is a cheek or the like and the accuracy of the motion vectors of the small regions included in the region of the specific part is relatively low because the values of each pixel included in the region of the specific part are relatively close, the accuracy of estimating the position of the destination of the specific part can be improved.

[0047] In this case, the estimation unit 12 of the information processing apparatus 10 may detect, for example, the region of a predetermined part of the subject in the first frame by image recognition or the like. The estimation unit 12 of the information processing apparatus 10 may calculate, for example, a vector indicating the relative position between each small region included in the region of a predetermined part of the subject in the first frame and each small region included in the region of the specific part. Then, the estimation unit 12 of the information processing apparatus 10 may calculate, for example, from the positions of each small region included in the region of the predetermined part in the first frame, the positions moved in the direction and by the amount of movement indicated by the motion vectors of the respective small regions. Then, the estimation unit 12 of the information processing apparatus 10 may calculate, for example, each of the calculated positions as the position of each small region in the second frame after moving in the direction and by the amount of movement indicated by the vector indicating the relative position described above. Then, the estimation unit 12 of the information processing apparatus 10 may estimate, for example, each of the calculated positions as the position of each small region in the second frame.

[0048] FIG. 9 illustrates examples of motion vectors 704A to 704D of each small region 703A to 703D included in the cheek region 702, which is a specific part, in a part 701 of the first frame, similar to FIG. 7. Also illustrated are examples of a motion vector 904A of a small region 903A included in the eye region, which is a predetermined part, in the first frame, and a motion vector 904B of a small region 903B included in the nose region, which is a predetermined part. Also illustrated are examples of a vector 905A indicating the relative position from the small region 903A to the small region 703A and a vector 905B indicating the relative position from the small region 903B to the small region 703A in the first frame.

[0049] The estimation unit 12 of the information processing apparatus 10 may estimate, for example, the position obtained by adding (combining) the motion vector 904A and the vector 905A from the position of the small region 903A as the position of the small region 703A in the second frame. Also, the estimation unit 12 of the information processing apparatus 10 may estimate, for example, the position obtained by adding the motion vector 904B and the vector 905B from the position of the small region 903B as the position of the small region 703A in the second frame. Also, the estimation unit 12 of the information processing apparatus 10 may estimate, for example, an average value or the like of the position obtained by adding the motion vector 904A and the vector 905A from the position of the small region 903A and the position obtained by adding the motion vector 904B and the vector 905B from the position of the small region 903B as the position of the small region 703A in the second frame. Also, the estimation unit 12 of the information processing apparatus 10 may estimate a representative value (for example, an average value, a mode value, a median value) of the values obtained by adding each vector indicating the relative position to each motion vector from the positions of a plurality of small regions included in one or more predetermined regions as the position of the small region of the specific part in the second frame.

[0050] (Example of setting image quality according to enlargement / reduction) The information processing apparatus 10 may estimate a change in the size on the pixel coordinates of the region of a specific part of the subject in the second frame based on the motion vector, and set encoding parameters based on the estimated change in the size of the region of the specific part for the second frame. Thereby, for example, even when the distance between the subject and the imaging apparatus 20 changes, an image of the specific part can be distributed with appropriate image quality. For example, when the subject approaches the imaging apparatus 20, an increase in the usage amount of the bandwidth of the network N can be reduced. Also, for example, when the subject moves away from the imaging apparatus 20, a decrease in the accuracy of analysis at the distribution destination or the like can be reduced.

[0051] In this case, the estimation unit 12 of the information processing apparatus 10 may detect, for example, that the size of the region of the specific part has changed based on the direction of the motion vector of each small region included in the region of the specific part. In this case, for example, when the directions of the motion vectors of each small region included in the edge of the region of the specific part are distributed so as to spread from the center of the region of the specific part, the estimation unit 12 of the information processing apparatus 10 may determine that the size of the region of the specific part is expanding. Also, for example, when the directions of the motion vectors of each small region included in the edge of the region of the specific part are distributed so as to go toward the center of the region of the specific part, the estimation unit 12 of the information processing apparatus 10 may determine that the size of the region of the specific part is shrinking.

[0052] Further, the estimation unit 12 of the information processing apparatus 10 may calculate, for example, the variance of the directions of the motion vectors of each small region included in the region of a specific part, and estimate the degree of change (expansion rate, reduction rate) of the size of the region of the specific part based on the calculated value. Then, when the region of the specific part is expanded in the second frame compared to the first frame, the control unit 13 of the information processing apparatus 10 may reduce the image quality of the region of the specific part as the expansion rate is higher. Also, when the region of the specific part is reduced in the second frame compared to the first frame, the control unit 13 of the information processing apparatus 10 may increase the image quality of the region of the specific part as the reduction rate is higher. Note that the control unit 13 of the information processing apparatus 10 can increase the image quality, for example, by setting at least one of the encoding bit rate and the frame rate high (large). Also, the control unit 13 of the information processing apparatus 10 can increase the image quality, for example, by reducing the quantization parameter (QP value) of the encoding.

[0053] Subsequently, the control unit 13 of the information processing apparatus 10 sets the encoding parameter for the second frame (step S105). The control unit 13 of the information processing apparatus 10 sets (determines) the encoding parameter for encoding the region of the specific part of the subject in the estimated second frame with a specific image quality.

[0054] Subsequently, the control unit 13 of the information processing apparatus 10 causes the second frame to be encoded with the set encoding parameter (step S106). Note that this process may be the same as the process of step S101. Thereby, it is possible to increase the image quality of the region used for inspection or the like in the second frame as in the first frame.

[0055] Subsequently, the control unit 13 of the information processing apparatus 10 causes the encoded second frame to be distributed (transmitted) to the destination device 30 via the network N (step S107). Note that this process may be the same as the process of step S102.

[0056] Subsequently, the receiving device 30 analyzes the information of the subject based on the area of the specific part with specific image quality in the received video (step S108). In the example of FIG. 10, in the decoded image 1001 of the second frame, at least a part of the area 802 of the cheek of the subject 1002 is received with a specific image quality.

[0057] Here, the receiving device 30 may measure (calculate, infer, estimate) information on various analysis targets of the subject by using, for example, AI (Artificial Intelligence) such as deep learning. The analysis targets may include, for example, at least one of heart rate, respiratory rate, blood pressure, swelling, percutaneous arterial oxygen saturation, pupil size, throat swelling, and periodontal disease degree. Note that the analysis targets may be specified (selected, set) in advance by a doctor or the like. Further, the receiving device 30 may determine one or more analysis targets based on the result of a medical interview input in advance from the patient by a predetermined website or the like.

[0058] The receiving device 30 may estimate the heart rate based on the video of the area where the patient's skin is exposed (for example, the face area). In this case, the receiving device 30 may estimate the heart rate based on, for example, the transition (period) of the change in skin color.

[0059] Further, the receiving device 30 may estimate the respiratory rate based on the video of the area of the patient's chest (upper body). In this case, the receiving device 30 may estimate the respiratory rate based on, for example, the period of the movement of the shoulders.

[0060] Further, the receiving device 30 may estimate the blood pressure based on the video of the area where the patient's skin is exposed (for example, the face area). In this case, the receiving device 30 may estimate the blood pressure based on, for example, the difference and shape of the pulse waves estimated from two locations on the face (for example, the forehead and the cheek).

[0061] Further, the destination device 30 may estimate the percutaneous arterial oxygen saturation (SpO2) based on an image of an area where the patient's skin is exposed (for example, the facial area). Note that red easily transmits when hemoglobin is combined with oxygen, and blue is less affected by the combination of hemoglobin and oxygen. Therefore, the destination device 30 may estimate SpO2 based on, for example, the difference in the degree of change between the blue and red colors of the skin near the cheekbone under the eyes.

[0062] Further, the destination device 30 may estimate the degree of swelling based on, for example, an image of the eyelid area of the patient. Further, the destination device 30 may estimate the pupil size (pupil diameter) based on, for example, an image of the eye area of the patient. Further, the destination device 30 may estimate the degree of throat swelling, periodontal disease, etc. based on, for example, an image of the area inside the patient's mouth.

[0063] (Example of remotely monitoring a vehicle using an image from the imaging device 20)

[0064] In the above-described example, an example was described in which, in a video conference between a doctor and a patient, the area of a specific part of the patient delivered in high quality is tracked, and visual inspection and biological information measurement are performed. Hereinafter, an example of remotely monitoring a vehicle using an image from the imaging device 20, which is a monitoring camera, will be described.

[0065] The estimation unit 12 of the information processing device 10 may first detect the area of a characteristic part of each vehicle by image recognition. Here, the estimation unit 12 of the information processing device 10 may extract, as the characteristic part, a part with a large change in luminance, such as a wheel, a window, a door, or advertisement characters. Then, the estimation unit 12 of the information processing device 10 may track each vehicle based on the small areas and motion vectors included in the area of the characteristic part of each vehicle in the first frame of the imaging device 20. Then, the control unit 13 of the information processing device 10 may set the image quality of the area to be higher than the image quality of other areas. Thereby, for example, an area such as a vehicle or surrounding vehicles photographed by a camera installed at a vehicle or an intersection can be delivered with a higher image quality than other areas.

[0066] In addition, for large vehicles such as buses and trucks, the amount of change in the pixel values on the side of the vehicle is relatively small in each frame. Therefore, in many cases, the actual movement vector (quantity and direction) of each small area does not match the movement vector calculated during encoding. On the other hand, according to the present disclosure, since tracking is performed using the movement vectors of the small areas included in the area of the characteristic part, tracking can be performed with higher accuracy.

[0067] (Example of remotely monitoring a ship using an image from the imaging device 20) Hereinafter, an example of remotely monitoring a ship using an image from the imaging device 20, which is a monitoring camera, will be described. The estimation unit 12 of the information processing device 10 may first detect the area of the characteristic part of each ship by image recognition. Here, the estimation unit 12 of the information processing device 10 may extract, as the characteristic part, a part with a large change in luminance, such as a bridge, a chimney, a mast, a window, a ship name display, etc. Then, the estimation unit 12 of the information processing device 10 may track each ship based on the small areas and movement vectors included in the area of the characteristic part of each ship in the first frame of the imaging device 20. Then, the control unit 13 of the information processing device 10 may set the image quality of the area to be higher than that of other areas. Thereby, for example, an area such as a ship in the vicinity photographed by a camera installed on a ship or in a harbor can be distributed with higher image quality than other areas.

[0068] In addition, for large ships such as tankers, the amount of change in the pixel values on the side of the ship is relatively small in each frame. Therefore, in many cases, the actual movement vector (quantity and direction) of each small area does not match the movement vector calculated during encoding. On the other hand, according to the present disclosure, since tracking is performed using the movement vectors of the small areas included in the area of the characteristic part, tracking can be performed with higher accuracy.

[0069] (Example of identifying a person using an image from the imaging device 20, which is a monitoring camera) Hereinafter, an example of identifying a person using an image from the imaging device 20, which is a monitoring camera, will be described.

[0070] The information processing apparatus 10 may track the area of a person based on a small area of the first frame of the imaging apparatus 20 and the motion vector, and may make the image quality of the area higher than that of other areas.

[0071] (Example of inspecting a product using an image of the imaging apparatus 20)

[0072] Hereinafter, an example of inspecting a product using an image of the imaging apparatus 20, which is a surveillance camera, will be described.

[0073] The information processing apparatus 10 may track the area of a specific part of a product based on a small area of the first frame of the imaging apparatus 20 and the motion vector, and may make the image quality of the area higher than that of other areas.

[0074] (Example of inspecting a facility using an image of the imaging apparatus 20) Hereinafter, an example of inspecting a facility using an image of the imaging apparatus 20 mounted on a drone or a robot that autonomously moves on the ground will be described. In this case, the video of the imaging apparatus 20 may be distributed from the information processing apparatus 10 mounted on the drone or the like to the destination apparatus 30.

[0075] The information processing apparatus 10 may track the area of an object to be inspected (for example, a steel tower, electric wires, etc.) based on a small area of the first frame of the imaging apparatus 20 and the motion vector, and may make the image quality of the area higher than that of other areas.

[0076] <Modification example> The information processing apparatus 10 may be a device included in one housing, but the information processing apparatus 10 of the present disclosure is not limited thereto. Each part of the information processing apparatus 10 may be realized by cloud computing configured by, for example, one or more computers. Further, at least a part of the processing of the information processing apparatus 10 may be realized by, for example, another information processing apparatus 10. Such information processing apparatuses 10 are also included in an example of the "information processing apparatus" of the present disclosure.

[0077] Note that the present disclosure is not limited to the above-described embodiments, and can be appropriately modified without departing from the gist thereof.

[0078] Some or all of the above-described embodiments can be described as follows in the appended claims, but are not limited thereto. (Appended Claim 1) For a frame of video distributed via a network, acquisition means for acquiring information indicating a movement vector for each of a plurality of small regions obtained by dividing the frame of the video into a plurality of regions; Estimation means for estimating the position of a region of a specific part of a subject in the frame of the video based on the information indicating the movement vector obtained by the acquisition means; Control means for setting encoding parameters for the frame of the video based on the position of the region of the specific part estimated by the estimation means; An information processing system having the above. (Appended Claim 2) The information indicating the movement vector includes a motion vector when encoding video using inter-frame prediction. The information processing system according to Appended Claim 1. (Appended Claim 3) The control means sets encoding parameters in which at least one of the encoding bit rate, frame rate, and encoding quantization parameter (QP value) for each small region included in the region of the specific part in the frame is a specific value. The information processing system according to Appended Claim 1 or 2. (Appended Claim 4) The estimation means estimates the position of the region of the specific part in the second frame based on the information indicating the movement vector from the position of each of the small regions included in the region of the specific part of the subject in the first frame to the position of each of the small regions included in the region of the specific part in the second frame. The information processing system according to any one of Appended Claims 1 to 3. (Appended Claim 5) The estimation means estimates the position of the region of the specific part of the subject in the fourth frame based on information indicating a vector of movement from the position of each small region included in the region of the predetermined part of the subject in the third frame to the position of each small region included in the region of the predetermined part of the subject in the fourth frame. The information processing system according to any one of Appendices 1 to 4. (Appendix 6) The estimation means estimates a change in the size of the region of the specific part of the subject in the frame based on the information indicating the vector of movement acquired by the acquisition means. The control means causes the encoding parameter based on the change in the size of the region of the specific part estimated by the estimation means to be set for the frame. The information processing system according to any one of Appendices 1 to 5. (Appendix 7) For a frame of video distributed via a network, a process of acquiring information indicating a movement vector for each of small regions obtained by dividing the frame of the video into a plurality of regions, A process of estimating the position of the region of the specific part of the subject in the frame of the video based on the information indicating the movement vector acquired in the acquiring process, A process of causing an encoding parameter to be set for the frame of the video based on the position of the region of the specific part estimated in the estimating process, An information processing method for executing the above. (Appendix 8) The information indicating the movement vector includes a motion vector when encoding video using inter-frame prediction. The information processing method according to Appendix 7. (Appendix 9) In the setting process, an encoding parameter is set such that at least one of the encoding bit rate, frame rate, and encoding quantization parameter (QP value) for each small region included in the region of the specific part in the frame is a specific value. The information processing method according to Appendix 7 or 8. (Appendix 10) In the estimating process, based on information indicating vectors of movement from the respective positions of each small region included in the region of a specific part of the subject in the first frame to the respective positions of each small region included in the region of the specific part in the second frame, the position of the region of the specific part in the second frame is estimated. An information processing method according to any one of Appendices 7 to 9. (Appendix 11) In the estimating process, based on information indicating vectors of movement from the respective positions of each small region included in the region of a predetermined part of the subject in the third frame to the respective positions of each small region included in the region of the predetermined part in the fourth frame, the position of the region of the specific part in the fourth frame is estimated. An information processing method according to any one of Appendices 7 to 10. (Appendix 12) In the estimating process, based on the information indicating the vectors of movement obtained in the obtaining process, a change in the size of the region of a specific part of the subject in the frame is estimated. The setting process causes the frame to be set with encoding parameters based on the change in the size of the region of the specific part estimated in the estimating process. An information processing method according to any one of Appendices 7 to 11. (Appendix 13) For a frame of video distributed via a network, acquisition means for acquiring information indicating vectors of movement for each of small regions obtained by dividing the frame of the video into a plurality of regions; Estimation means for estimating the position of the region of a specific part of the subject in the frame of the video based on the information indicating the vectors of movement acquired by the acquisition means; Control means for causing the frame of the video to be set with encoding parameters based on the position of the region of the specific part estimated by the estimation means; An information processing apparatus having the above. (Appendix 14) The information indicating the vectors of movement includes motion vectors when encoding video using inter-frame prediction. The information processing apparatus described in Supplementary Note 13. (Supplementary Note 15) The control means sets an encoding parameter in which at least one of the bit rate, frame rate, and quantization parameter (QP value) of encoding for each small region included in the region of the specific part in the frame is a specific value. The information processing apparatus described in Supplementary Note 13 or 14. (Supplementary Note 16) Based on the information indicating the vector of movement from the position of each small region included in the region of the specific part of the subject in the first frame to the position of each small region included in the region of the specific part in the second frame, the estimation means estimates the position of the region of the specific part in the second frame. The information processing apparatus according to any one of Supplementary Notes 13 to 15. (Supplementary Note 17) Based on the information indicating the vector of movement from the position of each small region included in the region of the predetermined part of the subject in the third frame to the position of each small region included in the region of the predetermined part in the fourth frame, the estimation means estimates the position of the region of the specific part in the fourth frame. The information processing apparatus according to any one of Supplementary Notes 13 to 16. (Supplementary Note 18) Based on the information indicating the vector of movement acquired by the acquisition means, the estimation means estimates the change in the size of the region of the specific part of the subject in the frame. Based on the change in the size of the region of the specific part estimated by the estimation means, the control means sets an encoding parameter for the frame. The information processing apparatus according to any one of Supplementary Notes 13 to 17.

Explanation of Signs

[0079] 1 Information processing system 10 Information processing apparatus 10A Information processing apparatus 10B Information processing apparatus 11 Acquisition unit 12 Estimation unit 13 Control Unit 20 Imaging Device N Network

Claims

1. An acquisition means for acquiring information indicating a vector of movement for each of small regions obtained by dividing a first frame of video distributed via a network into a plurality of regions; An estimation means for estimating the position of a region of a specific part, which is a subject to be analyzed, in a second frame of the video based on the information indicating the vector of movement acquired by the acquisition means; A control means for setting encoding parameters for the second frame of the video based on the position of the region of the specific part estimated by the estimation means; comprising: The estimation means calculates a vector indicating a relative position between each small region included in a region of a characteristic part that constitutes a subject in the first frame and different from the specific part and each small region included in the region of the specific part, and from the respective positions of each small region included in the region of the characteristic part in the first frame to the respective positions of each small region included in the region of the characteristic part in the second frame. Based on the information indicating the movement vector and the vector indicating the relative position, the position of the region of the specific part in the second frame is estimated. An information processing system.

2. The information indicating the vector of movement includes a motion vector when encoding video using inter-frame prediction. The information processing system according to claim 1.

3. The control means sets encoding parameters in which at least one of the encoding bit rate, frame rate, and encoding quantization parameter (QP value) for each small region included in the region of the specific part in the second frame is set to a specific value. The information processing system according to claim 1 or 2.

4. The estimation means estimates the position of the region of the specific part in the second frame based on the information indicating the vector of movement from the respective positions of each small region included in the region of the specific part in the first frame to the respective positions of each small region included in the region of the specific part in the second frame. The information processing system according to any one of claims 1 to 3.

5. Based on the information indicating the vectors of movement from the respective positions of each small region included in the region of the predetermined part of the subject in the third frame to the respective positions of each small region included in the region of the predetermined part in the fourth frame, the estimating means estimates the position of the region of the specific part in the fourth frame. The information processing system according to any one of claims 1 to 4.

6. Based on the information indicating the vector of movement obtained by the obtaining means, the estimating means estimates a change in the size of the region of the specific part of the subject in the frame. Based on the change in the size of the region of the specific part estimated by the estimating means, the control means causes encoding parameters to be set for the frame. The information processing system according to any one of claims 1 to 5.

7. A process of obtaining information indicating vectors of movement for each of a plurality of small regions obtained by dividing a first frame of video distributed via a network, A process of estimating the position of the region of the specific part, which is the object to be analyzed, of the subject in the second frame of the video based on the information indicating the vector of movement obtained in the obtaining process, A process of causing encoding parameters to be set for the second frame of the video based on the position of the region of the specific part estimated in the estimating process, is executed, In the estimating process, vectors indicating the relative positions between each small region included in the region of the characteristic part that constitutes the subject in the first frame and different from the specific part and each small region included in the region of the specific part are calculated, and based on the information indicating the vectors of movement from the respective positions of each small region included in the region of the characteristic part in the first frame to the respective positions of each small region included in the region of the characteristic part in the second frame and the vectors indicating the relative positions, the position of the region of the specific part in the second frame is estimated. An information processing method that executes the above.

8. The information indicating the vector of movement includes a motion vector when encoding video using inter-frame prediction. The information processing method according to claim 7.

9. In the process of setting, an encoding parameter is set such that at least one of the bit rate, frame rate, and quantization parameter (QP value) of encoding for each small region included in the region of the specific part in the second frame is a specific value. The information processing method according to claim 7 or 8.

10. In the process of estimating, based on the information indicating the vector of movement from each position of each small region included in the region of the specific part in the first frame to each position of each small region included in the region of the specific part in the second frame, the position of the region of the specific part in the second frame is estimated. The information processing method according to any one of claims 7 to 9.

11. In the process of estimating, based on the information indicating the vector of movement from each position of each small region included in the region of a predetermined part of the subject in the third frame to each position of each small region included in the region of the predetermined part in the fourth frame, the position of the region of the specific part in the fourth frame is estimated. The information processing method according to any one of claims 7 to 10.

12. In the process of estimating, based on the information indicating the vector of movement acquired in the process of acquiring, the change in the size of the region of the specific part of the subject in the frame is estimated. The process of setting causes the encoding parameter based on the change in the size of the region of the specific part estimated in the process of estimating to be set for the frame. The information processing method according to any one of claims 7 to 11.

13. An acquisition means for acquiring information indicating a vector of movement for each of a plurality of small regions obtained by dividing a first frame of a video distributed via a network; An estimation means for estimating the position of the region of the specific part, which is the object to be analyzed, of the subject in the second frame of the video based on the information indicating the vector of movement acquired by the acquisition means; A control means for setting an encoding parameter for the second frame of the video based on the position of the region of the specific part estimated by the estimation means; comprising. The estimation means calculates a vector indicating the relative positions between each small region included in a characteristic region that constitutes the subject in the first frame and that is different from the specific region, and each small region included in the region of the specific region, and based on information indicating a vector of movement from the position of each small region included in the region of the characteristic part in the first frame to the position of each small region included in the region of the characteristic part in the second frame, and the vector indicating the relative position, estimates the position of the region of the specific part in the second frame. An information processing apparatus.

14. The information indicating the movement vector includes a motion vector when encoding video using inter-frame prediction. The information processing apparatus according to claim 13.

15. The control means causes to set an encoding parameter in which at least one of the encoding bit rate, frame rate, and encoding quantization parameter (QP value) for each small region included in the region of the specific part in the second frame is a specific value. The information processing apparatus according to claim 13 or 14.

16. The estimation means estimates the position of the region of the specific part in the second frame based on information indicating a vector of movement from the position of each small region included in the region of the specific part in the first frame to the position of each small region included in the region of the specific part in the second frame. The information processing apparatus according to any one of claims 13 to 15.

17. The estimation means estimates the position of the region of the specific part in the fourth frame based on information indicating a vector of movement from the position of each small region included in the region of a predetermined part of the subject in the third frame to the position of each small region included in the region of the predetermined part in the fourth frame. The information processing apparatus according to any one of claims 13 to 16.

18. The estimation means estimates a change in the size of the region of a specific part of the subject in a frame based on the information indicating the movement vector acquired by the acquisition means. The control means causes to set an encoding parameter based on the change in the size of the region of the specific part estimated by the estimation means for the frame. The information processing apparatus according to any one of claims 13 to 17.

Images