Image processing system, image processing method, image processing device and program

Abstract

A video processing system (10) comprising: an image quality control unit (11) that controls the image quality of an gaze area including a gaze target in an inputted video; a recognition unit (12) that performs recognition processing on the video in which the the image quality of the gaze area has been controlled, said recognition processing recognizing the gaze target; a prediction unit (13) that, on the basis of extracted information which was extracted from the recognition processing, predicts the location of the gaze target in a later video that is after the video which was subjected to recognition processing; and a determination unit (14) that, on the basis of the predicted location of the gaze target, determines an gaze area in which the image quality control unit (11) will control the image quality in the later video.

Description

Technical Field

[0001] The present disclosure relates to a video processing system, a video processing method, and a video processing apparatus.

Background Art

[0002] The development of a system for performing monitoring and the like by applying a detection technique or a recognition technique using machine learning to video captured by a camera is in progress.

[0003] As a related technique, for example, Patent Document 1 is known. Patent Document 1 describes a technique for allocating a bandwidth to each camera according to the available bandwidth of a network and the importance of an object detected by each camera in a remote monitoring system that transmits video captured by a plurality of cameras mounted on a vehicle via the network. Patent Document 1 also describes predicting the position of an object and acquiring an area where the object may exist.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, it is possible to appropriately control the bandwidth for transmitting video according to the importance of an object detected from the video. On the other hand, in a system for performing recognition processing such as action recognition on video, it is desired to improve the recognition accuracy.

[0006] In view of such problems, an object of the present disclosure is to provide a video processing system, a video processing method, and a video processing apparatus capable of improving recognition accuracy.

Means for Solving the Problems

[0007] The video processing system according to this disclosure includes: image quality control means for controlling the image quality of a gaze region including a gaze object in an input video; recognition means for performing a recognition process to recognize the gaze object in a video in which the image quality of the gaze region has been controlled; prediction means for predicting the position of the gaze object in a video after the video in which the recognition process was performed, based on extracted information extracted from the recognition process; and determination means for determining the gaze region in which the image quality control means controls the image quality in the subsequent video, based on the predicted position of the gaze object.

[0008] The video processing method relating to this disclosure controls the image quality of a gaze region including a gaze target in an input video, performs a recognition process to recognize the gaze target in the video in which the image quality of the gaze region has been controlled, predicts the position of the gaze target in a video after the video in which the recognition process was performed based on the extracted information extracted from the recognition process, and determines the gaze region in which the image quality will be controlled in the subsequent video based on the predicted position of the gaze target.

[0009] The video processing device according to this disclosure includes: image quality control means for controlling the image quality of a gaze region including a gaze object in an input video; recognition means for performing a recognition process to recognize the gaze object in a video in which the image quality of the gaze region has been controlled; prediction means for predicting the position of the gaze object in a video after the video in which the recognition process was performed, based on extracted information extracted from the recognition process; and determination means for determining the gaze region in which the image quality control means controls the image quality in the subsequent video, based on the predicted position of the gaze object. [Effects of the Invention]

[0010] This disclosure provides an image processing system, an image processing method, and an image processing device that can improve recognition accuracy. [Brief explanation of the drawing]

[0011] [Figure 1]This is a configuration diagram showing an overview of the video processing system according to the embodiment. [Figure 2] This is a configuration diagram showing an overview of the video processing device according to the embodiment. [Figure 3] This is a flowchart outlining the video processing method according to the embodiment. [Figure 4] This is a diagram showing the basic configuration of a remote monitoring system. [Figure 5] This is a configuration diagram showing an example of the terminal configuration according to Embodiment 1. [Figure 6] This is a configuration diagram showing an example of the configuration of the central server according to Embodiment 1. [Figure 7] This is a configuration diagram showing an example of the configuration of the behavior recognition unit according to Embodiment 1. [Figure 8] This is a diagram showing another configuration example of the behavior recognition unit according to Embodiment 1. [Figure 9] This flowchart shows an example of the operation of the remote monitoring system according to Embodiment 1. [Figure 10] This is a diagram illustrating the video acquisition process according to Embodiment 1. [Figure 11] This is a diagram illustrating the object detection process according to Embodiment 1. [Figure 12] This flowchart shows an example of the operation of the action recognition process according to Embodiment 1. [Figure 13] This is a diagram illustrating the action recognition process according to Embodiment 1. [Figure 14] This is a diagram illustrating the action recognition process according to Embodiment 1. [Figure 15] This flowchart shows another example of the operation of the action recognition process according to Embodiment 1. [Figure 16] This is a diagram illustrating the action recognition process according to Embodiment 1. [Figure 17] This is a diagram illustrating the gaze target position prediction process according to Embodiment 1. [Figure 18] This is a diagram illustrating the gaze target position prediction process according to Embodiment 1. [Figure 19]This is a diagram for explaining the fixation target position prediction process according to Embodiment 1. [Figure 20] This is a diagram for explaining the fixation area determination process according to Embodiment 1. [Figure 21] This is a block diagram showing a configuration example of a terminal according to Embodiment 2. [Figure 22] This is a flowchart showing an operation example of a remote monitoring system according to Embodiment 2. [Figure 23] This is a flowchart showing an operation example of the matching determination process according to Embodiment 2. [Figure 24] This is a diagram for explaining the matching determination process according to Embodiment 2. [Figure 25] This is a block diagram showing an overview of the hardware of a computer according to the embodiment.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments will be described with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations are omitted as necessary.

[0013] In a system that collects video via a network and recognizes objects, actions, etc. in the video, since the bandwidth of the network for transmitting the video is limited, it is preferable to suppress the data amount of the video to be transmitted as much as possible. For example, the data amount of the video can be suppressed by increasing the compression rate of the video. However, when the compression rate of the video is high or the data loss rate is high, misrecognition increases, resulting in a decrease in recognition accuracy. Therefore, in the embodiment, it is possible to prevent misrecognition while suppressing the data amount of the video to be transmitted as much as possible.

[0014] (Overview of the Embodiment) First, the overview of the embodiment will be described. FIG. 1 shows an overview configuration of a video processing system 10 according to the embodiment. The video processing system 10 is applicable to, for example, a remote monitoring system that collects video via a network and monitors the video.

[0015] As shown in Figure 1, the video processing system 10 includes an image quality control unit 11, a recognition unit 12, a prediction unit 13, and a determination unit 14.

[0016] The image quality control unit 11 controls the image quality of the gaze region, which includes the object being gazed upon, in the input video. For example, the image quality control unit 11 may enhance the image quality of the gaze region compared to other regions, i.e., make it sharper. The recognition unit 12 performs a recognition process to recognize the object being gazed upon in the video in which the image quality of the gaze region has been controlled by the image quality control unit 11. The recognition process may be, for example, an action recognition process that recognizes the actions of the object being gazed upon, but it may also be a process that recognizes other information or characteristics of the object being gazed upon.

[0017] The prediction unit 13 predicts the position of the gazed object in video footage after the video footage in which the recognition processing was performed, based on information extracted from the recognition processing performed by the recognition unit 12. The extracted information is information about the extracted object extracted from the video by the video processing system 10. For example, the extracted information may include time-series position information of the gazed object, or it may include behavior recognition results, which are an example of the recognition results in the recognition processing. The determination unit 14 determines the gazed area in which the image quality control unit 11 controls the image quality in subsequent video footage, based on the position of the gazed object predicted by the prediction unit 13. The image quality control unit 11 controls the image quality of the gazed area determined by the determination unit 14 for the input video footage. For example, the image quality control unit 11 first controls the image quality according to a predetermined rule (for example, sharpening all areas), and then, after the prediction of the gazed object by the prediction unit 13 and the determination of the gazed area by the determination unit 14, it controls the image quality of the determined gazed area.

[0018] The video processing system 10 may consist of one device or multiple devices. Figure 2 shows the configuration of the video processing device 20 according to an embodiment. As shown in Figure 2, the video processing device 20 may include the image quality control unit 11, recognition unit 12, prediction unit 13, and determination unit 14 shown in Figure 1. Furthermore, part or all of the video processing system 10 may be located at the edge or in the cloud. For example, in a system that monitors video captured on site via a network, the edge refers to a device located at or near the site, and also a device closer to the terminal in the network hierarchy. For example, the image quality control unit 11 and determination unit 14 may be located at the edge terminal, and the recognition unit 12 and prediction unit 13 may be located on a cloud server. Furthermore, each function may be distributed across the cloud.

[0019] Figure 3 shows an image processing method according to an embodiment. For example, the image processing method according to the embodiment is performed by the image processing system 10 in Figure 1 and the image processing device 20 in Figure 2. As shown in Figure 3, first, the image quality of the gaze region, which includes the object of gaze, is controlled in the input image (S11). Next, a recognition process is performed to recognize the object of gaze in the image whose image quality of the gaze region has been controlled (S12). Next, based on the extracted information extracted from the recognition process, the position of the object of gaze in the image after the image in which the recognition process was performed is predicted (S13). Next, based on the predicted position of the object of gaze, the image quality control unit 11 determines the gaze region for which the image quality will be controlled in the subsequent image (S14). Furthermore, returning to S11, the image quality of the determined gaze region is controlled in the input image.

[0020] As described above, the video processing system according to the embodiment predicts the position of the object being watched in a later video based on the extracted information obtained from the recognition process performed on the video, and determines the area of ​​focus for which image quality is controlled in the later video based on the prediction result. This allows for the appropriate determination of the area for which image quality is controlled, thereby reducing the amount of video data, preventing misrecognition, and improving recognition accuracy.

[0021] (Basic configuration of a remote monitoring system) Next, we will describe a remote monitoring system, which is an example of a system to which the embodiment is applied. Figure 4 shows the basic configuration of the remote monitoring system 1. The remote monitoring system 1 is a system that monitors an area captured by a camera using the captured video. In this embodiment, it will be described as a system that remotely monitors the work of workers at a work site. For example, the work site may be a work site such as a construction site, a public square where people gather, a school, or any area where people or machines are operating. In this embodiment, the work will be described as construction work or civil engineering work, but it is not limited to these. Since video includes multiple images (also called frames) in a time series, video and images are interchangeable. That is, the remote monitoring system can be said to be a video processing system that processes video, and also an image processing system that processes images.

[0022] As shown in Figure 4, the remote monitoring system 1 includes multiple terminals 100, a central server 200, a base station 300, and an MEC 400. The terminals 100, base station 300, and MEC 400 are located on the field side, while the central server 200 is located on the central side. For example, the central server 200 is located in a data center or monitoring center located far from the field. The field side is the edge side of the system, and the central side is also the cloud side. The central server 200 may be composed of one device or multiple devices. Furthermore, some or all of the central server 200 may be located in the cloud. For example, the video recognition function 201 and the alert generation function 202 may be located in the cloud, while the GUI drawing function 203 and the screen display function 204 may be located in the monitoring center, etc.

[0023] Terminal 100 and base station 300 are connected via network NW1. Network NW1 is a wireless network such as 4G, local 5G / 5G, LTE (Long Term Evolution), or Wi-Fi. Base station 300 and central server 200 are connected via network NW2. Network NW2 includes core networks such as 5GC (5th Generation Core network) and EPC (Evolved Packet Core), as well as the internet. It can also be said that terminal 100 and central server 200 are connected via base station 300. Base station 300 and MEC 400 are connected via any communication method, but base station 300 and MEC 400 may be a single device.

[0024] Terminal 100 is a terminal device connected to the network NW1 and also a video generation device that generates on-site video. Terminal 100 acquires video captured by camera 101 installed on-site and transmits the acquired video to the center server 200 via base station 300. The camera 101 may be located outside or inside terminal 100.

[0025] Terminal 100 compresses the video from camera 101 to a predetermined bitrate and transmits the compressed video. Terminal 100 has a compression efficiency optimization function 102 and a video transmission function 103 that optimize the compression efficiency. The compression efficiency optimization function 102 performs ROI control, which controls the image quality of the ROI (Region of Interest; also called the area of ​​focus). The compression efficiency optimization function 102 reduces the bitrate by maintaining the image quality of the ROI, which includes people and objects, while lowering the image quality of the surrounding area. The video transmission function 103 transmits the video with controlled image quality to the central server 200.

[0026] Base station 300 is a base station device for network NW1 and also a relay device that relays communication between terminal 100 and central server 200. For example, base station 300 may be a local 5G base station, a 5G gNB (next generation node B), an LTE eNB (evolved node B), a wireless LAN access point, etc., but other relay devices may also be used.

[0027] The MEC (Multi-access Edge Computing) 400 is an edge processing unit located at the edge of the system. The MEC 400 is an edge server that controls terminals 100 and has a compressed bitrate control function 401 and a terminal control function 402 that control the bitrate of the terminals. The compressed bitrate control function 401 controls the bitrate of terminals 100 through adaptive video distribution control and QoE (quality of experience) control. For example, the compressed bitrate control function 401 predicts the recognition accuracy obtained while suppressing the bitrate according to the communication environment of networks NW1 and NW2, and allocates a bitrate to the camera 101 of each terminal 100 to improve the recognition accuracy. The terminal control function 402 controls terminals 100 to transmit video at the allocated bitrate. Terminals 100 encode the video to the allocated bitrate and transmit the encoded video.

[0028] The center server 200 is a server located on the central side of the system. The center server 200 may be one or more physical servers, or it may be a cloud server or other virtualized server built on the cloud. The center server 200 is a monitoring device that monitors on-site work by recognizing people's actions from on-site camera footage. The center server 200 is also a recognition device that recognizes the actions of people in the video transmitted from terminal 100.

[0029] The central server 200 has a video recognition function 201, an alert generation function 202, a GUI drawing function 203, and a screen display function 204. The video recognition function 201 inputs video transmitted from the terminal 100 into a video recognition AI (Artificial Intelligence) engine to recognize the type of work performed by the worker, i.e., the type of human action. The alert generation function 202 generates alerts according to the recognized work. The GUI drawing function 203 displays a GUI (Graphical User Interface) on the display device screen. The screen display function 204 displays video from the terminal 100, recognition results, alerts, etc. on the GUI.

[0030] (Embodiment 1) Embodiment 1 will be described below with reference to the drawings. First, the configuration of the remote monitoring system according to this embodiment will be described. The basic configuration of the remote monitoring system 1 according to this embodiment is as shown in Figure 4. Here, an example configuration of the terminal 100 and the center server 200 will be described. Figure 5 shows an example configuration of the terminal 100 according to this embodiment, and Figure 6 shows an example configuration of the center server 200 according to this embodiment. Note that the configuration of each device is just an example, and other configurations are also acceptable as long as the operation according to this embodiment described later is possible. For example, some functions of the terminal 100 may be placed in the center server 200 or other devices, and some functions of the center server 200 may be placed in the terminal 100 or other devices.

[0031] As shown in Figure 5, the terminal 100 includes a video acquisition unit 110, a detection unit 120, an image quality change determination unit 130, a compression efficiency determination unit 140, and a terminal communication unit 150.

[0032] The video acquisition unit 110 acquires video footage (also referred to as input video) captured by the camera 101. For example, the input video may include people, such as workers performing tasks on site, or the work objects used by those people (also referred to as work objects). The video acquisition unit 110 is also an image acquisition unit that acquires multiple images in a time series.

[0033] The detection unit 120 is an object detection unit that detects objects in the acquired input video. The detection unit 120 detects objects in each image included in the input video and assigns a label, i.e., an object label, to the detected object. The object label is the class of the object and indicates the type of object. The detection unit 120 extracts a rectangular region containing an object from each image included in the input video, recognizes the object within the extracted rectangular region, and assigns a label to the recognized object. The rectangular region is a bounding box or an object region. Note that the object region containing an object is not limited to a rectangular region, but may also be a circular or irregularly shaped silhouette region. The detection unit 120 calculates the image feature quantities of the object included in the rectangular region and recognizes the object based on the calculated feature quantities. For example, the detection unit 120 recognizes objects in the image using an object recognition engine that uses machine learning such as deep learning. By machine learning the image features of the object and the object label, the object can be recognized. The object detection result includes the object label, the position information of the rectangular region containing the object, etc. The object's position information is, for example, the coordinates of each vertex of a rectangular area, but it could also be the position of the center of the rectangular area, or the position of any point on the object. The detection unit 120 transmits the detection result of the object to the image quality change determination unit 130.

[0034] The image quality change determination unit 130 determines the gaze area (ROI), which is an image quality change region in the acquired input video, where the image quality is changed. The image quality change determination unit 130 is a determination unit that determines the gaze area. The gaze area is the region that includes the object being gazed at, and it is the region where the image quality is improved, that is, sharpened. It can also be said that the gaze area is the region that ensures image quality for action recognition.

[0035] For example, the image quality change determination unit 130 comprises a first determination unit 131 and a second determination unit 132. For example, first the first determination unit 131 determines the gaze area, and after the center server 200 recognizes the action, the second determination unit 132 determines the gaze area. Alternatively, the determination of the gaze area by the first determination unit 131 may be omitted, and only the determination of the gaze area by the second determination unit 132 may be performed. The first determination unit 131 determines the gaze area of ​​the input video based on the detection results of objects detected in the input video. The first determination unit 131 determines the gaze area based on the position information of objects with labels that are the object of gaze among the detected objects detected in the input video by the detection unit 120. The object of gaze is a person who is the subject of action recognition, but may also include work objects that the person can use in their work. For example, the labels of work objects are pre-set as labels of objects associated with the person.

[0036] The second determination unit 132 determines the gaze area of ​​the input video based on the information fed back from the central server 200 that has recognized the action. In this example, the information fed back is predicted information about the object of gaze. The predicted information about the object of gaze is information about the object of gaze, which is information predicted by the central server 200 through action recognition to determine the object of gaze in the next video. The predicted information about the object of gaze is information extracted from the predicted position of the object of gaze and the action recognition process, and includes position information of the rectangular area of ​​the object of gaze. For example, the second determination unit 132 determines the rectangular area indicated by the acquired predicted information as the gaze area. That is, it determines the area that ensures the image quality of the input video based on the predicted position of the object of gaze.

[0037] Furthermore, the prediction information obtained from the center server 200 may include the score of the behavior label, which is the result of behavior recognition. The second determination unit 132 may obtain the score of the behavior label, which is the result of behavior recognition, from the center server 200 and determine whether or not to determine a gaze area based on the obtained score. The score of the behavior label indicates the confidence level, which is the likelihood (probability) of the behavior label. The higher the score, the higher the probability that the predicted behavior label is correct. For example, if the score is smaller than a predetermined value, it is determined that it is necessary to ensure the image quality of the unrecognized area and to perform further behavior recognition, and the gaze area is determined based on the prediction information. If the score is larger than a predetermined value, it is determined that it is not necessary to perform further behavior recognition on the recognized area, and the gaze area is not determined. Conversely, if the score is larger than a predetermined value, it is determined that it is necessary to perform further behavior recognition on the recognized area, and the gaze area is determined based on the prediction information. If the score is smaller than a predetermined value, Kina If it is determined that further action recognition is not necessary for a particular area, the gaze area does not need to be determined. If the gaze area is not determined, the compression efficiency determination unit 140 does not need to enhance the image quality of the gaze area.

[0038] The compression efficiency determination unit 140 determines the compression ratio for the gaze region or other regions besides the gaze region, and compresses the video. The compression efficiency determination unit 140 is an encoder that encodes the input video according to the determined compression ratio. The compression efficiency determination unit 140 encodes using a video encoding scheme such as H.264 or H.265. The compression efficiency determination unit 140 also encodes the input video so that it matches the bitrate assigned by the MEC400's compression bitrate control function 401.

[0039] The compression efficiency determination unit 140 is an image quality control unit that controls the image quality of the gaze region determined by the image quality change determination unit 130, and is an image quality enhancement unit that enhances the image quality of the gaze region. The gaze region is the region determined by either the first determination unit 131 or the second determination unit 132. The compression efficiency determination unit 140 encodes the gaze region so that its image quality reaches a predetermined quality by compressing the gaze region and other regions at predetermined compression ratios. In other words, it enhances the image quality of the gaze region compared to other regions by changing the compression ratio of the gaze region and other regions. It can also be said that it reduces the image quality of other regions compared to the gaze region. For example, it controls the image quality of the gaze region and other regions within the range of bitrates allocated by the compression bitrate control function 401 of the MEC400. Note that the image quality of the gaze region may be controlled not only by the compression ratio, but also by changing the image resolution, frame rate, etc. Furthermore, the image quality of the gaze region may be controlled by changing the amount of color information in the image, for example, color, grayscale, black and white, etc.

[0040] The terminal communication unit 150 transmits the encoded data encoded by the compression efficiency determination unit 140 to the center server 200 via the base station 300. The terminal communication unit 150 is a transmission unit that transmits video with controlled image quality for the gaze area. The terminal communication unit 150 also receives prediction information of the gaze target transmitted from the center server 200 via the base station 300. The terminal communication unit 150 is an acquisition unit that acquires prediction information predicting the position of the gaze target. The terminal communication unit 150 is an interface that can communicate with the base station 300, and is a wireless interface such as 4G, local 5G / 5G, LTE, or Wi-Fi, but may also be a wireless or wired interface of any other communication method. The terminal communication unit 150 may include a first terminal communication unit that transmits encoded data and a second terminal communication unit that receives prediction information of the gaze target. The first terminal communication unit and the second terminal communication unit may be communication units of the same communication method or communication units of different communication methods.

[0041] As shown in Figure 6, the center server 200 includes a center communication unit 210, a decoder 220, an action recognition unit 230, an extracted information storage unit 240, a gaze target analysis unit 250, and a gaze target position prediction unit 260.

[0042] The central communication unit 210 receives encoded data transmitted from the terminal 100 via the base station 300. The central communication unit 210 is a receiving unit that receives video with controlled image quality for the gaze area. The central communication unit 210 also transmits predicted information about the gaze target predicted by the gaze target position prediction unit 260 to the terminal 100 via the base station 300. The central communication unit 210 is a notification unit that notifies the predicted information predicting the position of the gaze target. The central communication unit 210 is an interface that can communicate with the internet or a core network, for example, a wired interface for IP communication, but it may also be a wired or wireless interface of any other communication method. The central communication unit 210 may include a first central communication unit that receives encoded data and a second central communication unit that transmits predicted information about the gaze target. The first central communication unit and the second central communication unit may be communication units of the same communication method, or they may be communication units of different communication methods.

[0043] The decoder 220 decodes the encoded data received from the terminal 100. The decoder 220 corresponds to the encoding method of the terminal 100 and decodes using a video encoding method such as H.264 or H.265. The decoder 220 decodes according to the compression ratio of each region and generates the decoded video (also referred to as the received video).

[0044] The action recognition unit 230 recognizes the actions of objects in the decoded received video. The action recognition unit 230 performs action recognition processing to recognize the actions of the object being watched in video where the image quality of the gaze area has been controlled. The action recognition unit 230 detects objects from the received video and recognizes the actions of the detected objects. The action recognition unit 230 recognizes the actions of the person who is the target of action recognition and assigns a label to the recognized action, i.e., an action label. The action label is the class of the action and indicates the type of action.

[0045] For example, the action recognition unit 230 recognizes a person's actions based on the person and the work object detected from the received video. The action recognition unit 230 may also recognize a person's actions by identifying the relationship between the person and the work object. The relationship between the person and the work object includes which object the person is using or not using. For example, the work object may be identified for each person based on the distance between the person and the work object, and actions may be recognized from the identified work object. The person's actions may be recognized in a rule-based manner by associating work objects and tasks related to the person, or the person's actions may be recognized in a machine learning-based manner by machine learning the relationship between work objects and tasks related to the person.

[0046] The extracted information storage unit 240 stores the extracted information extracted by the action recognition processing of the action recognition unit 230. The extracted information includes action recognition results, person detection information, and detection information of work objects related to the action. The action recognition results include the label of the recognized action, the score of the action label, identification information of the person performing the recognized action, and identification information of the work object used in the recognized action. The person detection information includes the position information of the rectangular area of ​​the person, tracking information, etc. The tracking information is trajectory information that shows the tracking result of the object. The work object detection information includes the object label, the score of the object label, the position information of the rectangular area of ​​the object, tracking information, etc. For example, the action predictor (action recognition engine) of the action recognition unit 230 learns to assign weights to objects related to the action, extracts candidate work objects that may be related to each image, and outputs information on the extracted candidate work objects. For example, if pile driving work is recognized, it outputs information on a hammer, which is an object related to the action.

[0047] The gaze target analysis unit 250 determines the gaze target based on the extracted information extracted by the action recognition processing of the action recognition unit 230. The extracted information may be obtained from the action recognition unit 230 or from the extracted information storage unit 240. Based on the extracted information, the gaze target analysis unit 250 determines the gaze target in order to ensure image quality in order to prevent action recognition errors. For example, the gaze target analysis unit 250 determines the gaze target based on the action recognition result. The gaze target analysis unit 250 makes the person whose action has been recognized by the action recognition unit 230, that is, the person whose action is included in the action recognition result, the gaze target. If an action is recognized from a person and a work object associated with that person, the person and the work object may be made the gaze target. There may be multiple work objects associated with a person, and the person and multiple work objects may be made the gaze target. For example, if pile driving work is recognized, the objects associated with the work may be "stakes" and "hammers," and the person and the "stakes" and "hammers" may be made the gaze target.

[0048] The gaze target position prediction unit 260 predicts the position of the gaze target in the next video. The next video is the video after the video in which the action recognition processing was performed, and is the next video (input video) acquired by terminal 100. The next video is the video after a predetermined time has elapsed since the video in which the action was recognized. The timing of the next video, i.e., the prediction timing, is, for example, after the time has elapsed from when the video to be recognized is transmitted from terminal 100 until the prediction information is fed back to terminal 100 from the center server 200. The prediction timing of the next video may be determined by considering the transmission time between terminal 100 and the center server 200. For example, the prediction timing of the next video may be determined by measuring or acquiring the transmission time between terminal 100 and the center server 200.

[0049] The gaze target position prediction unit 260, based on the extracted information extracted by the action recognition processing of the action recognition unit 230, then... In video image quality ofThe gaze target position prediction unit 260 predicts the position of the gaze target to be secured. The gaze target position prediction unit 260 may predict the position of the gaze target based on the time-series position information of the person or work object whose action has been recognized. For example, the time-series position information is trajectory information obtained from the tracking process in the action recognition process. The gaze target position prediction unit 260 may also predict the position of the gaze target based on the action recognition result in which the action has been recognized. For example, the position of the gaze target may be predicted based on the work object (working object) used by the person in the action indicated by the action recognition result. The gaze target position prediction unit 260 predicts the position of the gaze target, taking into account the time difference until the next image. The gaze target position prediction unit 260 predicts the position of the gaze target and its rectangular area by moving the gaze target on the image according to the predicted timing of the next image. For example, the size and shape of the rectangular area may be changed according to the predicted timing of the next image to be predicted. The size of the rectangular area may be increased as the time until the prediction timing increases. The gaze target position prediction unit 260 outputs the position information of the predicted rectangular area of ​​the gaze target as the gaze target prediction information. Location information may be, for example, the coordinates of each vertex of a rectangular area, but it may also be the position of the center of the rectangular area, or the position of any point on the object being watched. Prediction information may include not only location information, but also information extracted from the action recognition process, such as the object label, image features, action label, and action label score of the object being watched. Furthermore, multiple prediction information may be output, such as information predicted from the time-series information of the recognized object or information predicted from the action recognition results. It may also be possible to predict the position at multiple points in time and output multiple predicted location information.

[0050] Figures 7 and 8 show examples of the configuration of the behavior recognition unit 230 in the central server 200. Figure 7 shows an example of a configuration when behavior recognition based on the relationship between a person and a work object is performed using a rule-based approach. In the example in Figure 7, the behavior recognition unit 230 includes an object detection unit 231, a tracking unit 232, a relationship analysis unit 233a, and a behavior determination unit 234.

[0051] The object detection unit 231 detects objects in the received video input. For example, the object detection unit 231 is a detection unit such as an object recognition engine using machine learning, similar to the detection unit 120 of the terminal 100. That is, the object detection unit 231 extracts a rectangular region containing an object from each image of the received video, recognizes the object within the extracted rectangular region, and assigns a label to the recognized object. The object detection result includes the object label and the location information of the rectangular region containing the object.

[0052] The tracking unit 232 tracks the detected objects in the received video. Based on the object detection results, the tracking unit 232 associates objects in each image included in the received video. By assigning a tracking ID to the detected objects, each object can be identified and tracked. For example, objects are tracked by associating them between images based on the distance or overlap (e.g., IoU: Intersection over Union) between the rectangular area of ​​an object detected in the previous image and the rectangular area of ​​an object detected in the next image.

[0053] The relationship analysis unit 233a analyzes the relationship between each tracked object and other objects. Specifically, the relationship analysis unit 233a analyzes the relationship between a person, who is the target of behavior recognition, and work objects that the person can use in their work. For example, the labels of work objects are pre-set as labels for objects associated with a person. For example, the relationship between objects is the position of the objects or the distance or overlap between rectangular areas (e.g., IoU). Based on the relationship between a person and a work object, it is possible to determine whether or not the person is using a work object to perform a task. For example, work objects associated with a person are extracted based on the distance or overlap between the person and the work object.

[0054] The behavior determination unit 234 determines the behavior of an object based on the relationships between the analyzed objects. The behavior determination unit 234 pre-associates work objects with work content and recognizes the work content of a person based on work objects related to the person extracted from the relationships between the person and work objects. The work content may also be recognized based on the person's characteristics, including their posture and shape, and related work objects. For example, the person's characteristics and work objects may be associated with the work content. The behavior determination unit 234 outputs the recognized work content of the person as a behavior label.

[0055] Furthermore, if no work object related to a person is detected, the action determination unit 234 may recognize the person's actions from the person alone. For example, the person's posture and shape may be pre-associated with the work content as characteristics of the person, and the work content may be identified based on the person's posture and shape extracted from the image.

[0056] Figure 8 shows an example configuration for machine learning-based behavior recognition based on the relationship between a person and a work object. In the example in Figure 8, the behavior recognition unit 230 includes an object detection unit 231, a tracking unit 232, a behavior predictor 233b, and a behavior determination unit 234. In this example, the behavior recognition unit 230 includes a behavior predictor 233b instead of the relationship analysis unit 233a in Figure 7, and the other configurations are the same as in Figure 7.

[0057] The behavior predictor 233b predicts the behavior of each object tracked by the tracking unit 232. The behavior predictor 233b recognizes the behavior of a person tracked in the received video and assigns a label to the recognized behavior. For example, the behavior predictor 233b recognizes the behavior of a person in the received video using a behavior recognition engine that employs machine learning, such as deep learning. By using machine learning on video of a person working with a work object and its behavior label, the behavior of a person can be recognized. For example, machine learning is performed using training data, which is video of a person working with a work object, annotation information such as the position of the person and the work object and the relationship between the person and the object, and behavior information such as the work object required for each task. The behavior predictor 233b also outputs a score for the recognized behavior label.

[0058] The behavior determination unit 234 determines the behavior of an object based on the predicted behavior label. The behavior determination unit 234 determines the behavior of a person based on the score of the behavior label predicted by the behavior predictor 233b. For example, the behavior determination unit 234 outputs the behavior label with the highest score as the recognition result.

[0059] Next, the operation of the remote monitoring system according to this embodiment will be described. Figure 9 shows an example of the operation of the remote monitoring system 1. For example, the explanation assumes that terminal 100 executes S101 to S105 and S112 to S113, and the center server 200 executes S106 to S111, but this is not limited to this, and any device may execute each process.

[0060] As shown in Figure 9, terminal 100 acquires video from camera 101 (S101). Camera 101 generates video of the site, and video acquisition unit 110 acquires the video output from camera 101 (input video). For example, as shown in Figure 10, the input video image includes people working at the site and work objects such as hammers used by the people.

[0061] Next, terminal 100 detects objects based on the acquired input video (S102). The detection unit 120 uses an object recognition engine to detect rectangular regions within the image contained in the input video, recognizes objects within the detected rectangular regions, and assigns labels to the recognized objects. For each detected object, the detection unit 120 outputs the object label and the position information of the object's rectangular region as the object detection result. For example, when object detection is performed on the image in Figure 10, as shown in Figure 11, a person and a hammer are detected, and the rectangular regions of the person and the hammer are detected.

[0062] Next, terminal 100 determines the gaze area in the input video based on the object detection results (S103). The first determination unit 131 of the image quality change determination unit 130 extracts objects that have a label to be gazed on based on the object detection results of each object. The first determination unit 131 extracts objects from the detected objects whose object label is a person or a work object, and determines the rectangular area of ​​the corresponding object as the gaze area. In the example in Figure 11, a person and a hammer are detected in the image, and the hammer is a work object, so the rectangular area of ​​the person and the rectangular area of ​​the hammer are determined as the gaze area.

[0063] Next, terminal 100 encodes the input video based on the determined gaze area (S104). The compression efficiency determination unit 140 encodes the input video so that the gaze area has higher image quality than other areas. In the example in Figure 11, the compression ratio of the rectangular area of ​​the person and the rectangular area of ​​the hammer is lowered compared to the compression ratio of other areas, thereby improving the image quality of the rectangular area of ​​the person and the rectangular area of ​​the hammer.

[0064] Next, terminal 100 transmits the encoded data to the central server 200 (S105), and the central server 200 receives the encoded data (S106). Terminal communication unit 150 transmits the encoded data, which has been enhanced in quality for the gaze area, to base station 300. Base station 300 forwards the received encoded data to the central server 200 via the core network or the internet. Central communication unit 210 receives the forwarded encoded data from base station 300.

[0065] Next, the center server 200 decodes the received encoded data (S107). The decoder 220 decodes the encoded data according to the compression ratio of each region and generates a video (received video) with the gaze region in high quality.

[0066] Next, the center server 200 recognizes the behavior of objects based on the decoded received video (S108). Figure 12 shows an example of the behavior recognition process by the behavior recognition unit 230 shown in Figure 7. In the example in Figure 12, first, the object detection unit 231 detects objects in the input received video (S201). The object detection unit 231 uses an object recognition engine to detect rectangular regions within each image included in the received video, recognizes objects within the detected rectangular regions, and assigns labels to the recognized objects. For each detected object, the object detection unit 231 outputs the object label and the position information of the object's rectangular region as the object detection result.

[0067] Next, the tracking unit 232 tracks the detected objects in the received video (S202). The tracking unit 232 assigns a tracking ID to each detected object and tracks the objects identified by the tracking ID in each image.

[0068] Next, the relationship analysis unit 233a analyzes the relationship between each tracked object and other objects (S203) and determines whether or not there are work objects related to a person (S204). The relationship analysis unit 233a extracts people and work objects from the detection results of the tracked objects and determines the distance and overlap of rectangular areas between the extracted person and work object. For example, work objects whose distance from a person is less than a predetermined value, or work objects whose overlap of the person's rectangular area is greater than a predetermined value, are determined to be work objects related to a person.

[0069] If it is determined that there is a work object associated with a person, the action determination unit 234 determines the person's action based on the person and the work object (S205). The action determination unit 234 determines the person's action based on the detected work object associated with the person and the work content that has been pre-associated with the work object. In the example in Figure 13, a person and a hammer associated with the person are detected by tracking. In addition, the work object-work content table is pre-stored as an association between work objects and work content. The work object-work content table is stored in the storage unit of the center server 200, etc. The action determination unit 234 refers to the work object-work content table from the work object associated with the person and identifies the work content associated with the work object. In this example, since driving a stake is associated with the hammer, the person's action is determined to be driving a stake. The action determination unit 234 outputs the determined action. For example, the relationship between the person and the work object (distance, overlap, etc.) may be output as the action score.

[0070] Furthermore, if it is determined that there are no work objects associated with the person, the action determination unit 234 determines the person's action based on the person (S206). The action determination unit 234 determines the person's action based on the characteristics of the detected person, such as their posture and shape, and the work content that has been pre-associated with the person's characteristics. In the example in Figure 14, only the person is detected by tracking. In addition, the posture-work content table is pre-associated and stored with the person's posture and work content. The posture-work content table is stored in the storage unit of the center server 200, etc. For example, the person's posture can be estimated based on the skeleton extracted from the person's image using a posture estimation engine. The action determination unit 234 estimates the person's posture from the detected person's image and refers to the posture-work content table to identify the work content associated with the estimated posture. In this example, if the estimated person's posture is posture B, since work B is associated with posture B, the person's action is determined to be work B. The action determination unit 234 outputs the determined action. For example, the score of the estimated person's posture may be output as the action score.

[0071] Furthermore, Figure 15 shows an example of action recognition processing by the action recognition unit 230 shown in Figure 8. In the example in Figure 15, similar to Figure 12, the object detection unit 231 detects an object in the received video (S201), and the tracking unit 232 tracks the detected object in the received video (S202).

[0072] Next, the behavior predictor 233b predicts the behavior of each tracked object (S207). The behavior predictor 233b uses an action recognition engine to predict the behavior of a person from the video including the tracked person and the work object. The behavior predictor 233b outputs the label of the predicted behavior and the score for each behavior label.

[0073] Next, the action determination unit 234 determines the action of the object based on the score of the predicted action label (S208). In the example in Figure 16, a person and a hammer are detected by tracking. The action predictor 233b recognizes the person's action based on the images of the detected person and hammer and outputs a score for each action label. For example, the score for pile driving is 0.8, the score for heavy machinery operation is 0.1, the score for unsafe behavior is 0.0, and the score for non-work is 0.1. In this case, the action determination unit 234 determines that the person's action is pile driving because the score for pile driving is the highest. The action determination unit 234 outputs the determined action and the score of the action.

[0074] Returning to Figure 9, following the action recognition process, the center server 200 determines the gaze target based on the extracted information obtained by the action recognition process (S109). The gaze target analysis unit 250 selects the person whose action has been recognized as the gaze target, and if the recognized target includes a work object, it also includes the work object as the gaze target. For example, in the examples of Figure 13 and Figure 16, the pile driving work is recognized from the person and the hammer, so the person and hammer that recognized the work are selected as the gaze targets. In the example of Figure 14, since work B is recognized only from the person, only the person that recognized the work is selected as the gaze target.

[0075] Next, the center server 200 predicts the position of the gaze target in the next video based on the extracted information obtained through the action recognition process (S110). The gaze target position prediction unit 260 uses the time-series information and action recognition results extracted during action recognition to predict the position (movement area) of the next gaze target, and outputs the position information of the predicted rectangular area of ​​the gaze target as the gaze target prediction information.

[0076] For example, when using time-series information, the gaze target position prediction unit 260 predicts the next movement area of ​​a person or work object from the trajectory information tracked by the person or work object. The trajectory information is acquired from the tracking unit 232 and may be acquired using a Kalman filter or particle filter. In the example in Figure 17, trajectory information of a person and a hammer is extracted from the video with recognized action. The gaze target position prediction unit 260 predicts the movement area based on the extension of the trajectory information. That is, the extension of the trajectory information is defined as the movement area. The gaze target position prediction unit 260 extends the trajectory information of the person or hammer on the image according to the predicted timing of the next video and predicts the position of the person or hammer's next movement area (rectangular area).

[0077] Furthermore, when using the action recognition results, the gaze target position prediction unit 260 determines the position (movement area) of the next gaze target for each action label using a rule-based approach. The movement area may also be predicted based on the orientation of the work object or person. For example, if excavation work is recognized, the direction the shovel or bucket is pointing may be used as the movement area. In the example in Figure 18, the person's action is recognized as excavation work, and information on the person and shovel is extracted. For example, the gaze target position prediction unit 260 recognizes the shape of the shovel, uses the direction of the tip of the shovel as the shovel's orientation, and extracts this shovel orientation as the excavation direction (work direction). The gaze target position prediction unit 260 moves the shovel or person in the excavation direction on the image according to the predicted timing of the next video, and predicts the position of the next movement area (rectangular area) of the shovel or person.

[0078] Furthermore, the position of a shovel or person can be predicted not only by using the orientation of the shovel, but also by using the orientation of the person. For example, the orientation of a person (forward direction) can be estimated from the skeleton and posture extracted from the person's image. This orientation of the person can be used as the digging direction to predict the movement area of ​​the shovel or person. Alternatively, the digging direction can be extracted by combining the orientation of the shovel and the orientation of the person.

[0079] Furthermore, for example, if compaction work is recognized, the area where the compactor is moving may be used as the movement area. In the example in Figure 19, the person's actions are recognized as compaction work, and information on the person and the compactor is extracted. For example, the gaze target position prediction unit 260 recognizes the shape of the compactor, sets the forward direction of the compactor as the direction of the compactor, and extracts this direction of the compactor as the compaction direction (work direction). The gaze target position prediction unit 260 moves the compactor and the person in the compaction direction on the image according to the predicted timing of the next video, and predicts the position of the next movement area (rectangular area) of the compactor and the person. Similar to Figure 18, the direction of the person may be used as the compaction direction, or the direction of the compactor and the direction of the person may be combined to extract the compaction direction.

[0080] Next, the center server 200 notifies the terminal 100 of the predicted information of the target of attention (S111), and the terminal 100 acquires the predicted information of the target of attention (S112). The center communication unit 210 transmits the predicted information indicating the location and area of ​​the predicted target of attention to the base station 300 via the internet or core network. The base station 300 forwards the received predicted information of the target of attention to the terminal 100. The terminal communication unit 150 receives the forwarded location information of the target of attention from the base station 300.

[0081] Next, terminal 100 determines the gaze area based on the received predicted information about the gaze target (S113). The second determination unit 132 of the image quality change determination unit 130 determines the area indicated by the predicted information about the gaze target notified by the center server 200 as the gaze area. In the example in Figure 20, the predicted information indicates the rectangular area of ​​the person and the rectangular area of ​​the hammer, and these areas are determined to be the gaze area. Alternatively, the bounding area including the rectangular area of ​​the person and the rectangular area of ​​the hammer may be designated as the gaze area. This bounding area may also be notified to terminal 100 from the center server 200. Steps S104 to S113 are then repeated.

[0082] As described above, in this embodiment, in a system that recognizes the behavior of an object from video, the system predicts the position of the object in the next video based on the object's time-series information and behavior recognition results, and enhances the image quality and clarity of the predicted area. This ensures that the image quality of a specific part including the object can be guaranteed according to the object's movement, while areas not related to behavior recognition can be compressed, thereby reducing the amount of data transmitted and preventing behavior recognition errors.

[0083] (Embodiment 2) Embodiment 2 will be described below with reference to the drawings. First, the configuration of the remote monitoring system according to this embodiment will be described. In this embodiment, only the configuration of the terminal differs from that of Embodiment 1, so here we will describe an example of the terminal configuration. Note that this embodiment can be implemented in combination with Embodiment 1, and each configuration shown in Embodiment 1 may be used as appropriate.

[0084] Figure 21 shows an example configuration of the terminal 100 according to this embodiment. As shown in Figure 21, in this embodiment, a matching unit 133 is added to the image quality change determination unit 130 of the terminal 100. The other configurations are the same as in Embodiment 1.

[0085] The matching unit 133 matches the predicted information of the object being watched, notified by the central server 200, with the detection results of the object detected by the detection unit 120 from the input video. In other words, it matches the object being watched predicted by the central server 200 with the object detected by the terminal 100. The input video in which the object to be matched is detected is video from after the video in which the central server 200 performed action recognition, that is, video corresponding to the predicted information of the object being watched predicted by the central server 200. The matching compares the predicted information of the object being watched with the detection results of the object to determine whether the predicted object and the detected object are the same, that is, whether or not they match. The matching unit 133 performs matching based on, for example, the type of object, the image characteristics of the object, the position information of the object, etc.

[0086] The second determination unit 132 determines the gaze area of ​​the input video based on the matching result of the matching unit 133. Depending on whether the predicted information of the gaze target matches the object detection result, the second determination unit 132 may determine the gaze area based on the object detection result or the predicted information of the gaze target, or it may determine whether or not to determine the gaze area.

[0087] Next, the operation of the remote monitoring system according to this embodiment will be described. Figure 22 shows an example of the operation of the remote monitoring system according to this embodiment. Steps S101 to S111 in Figure 22 are the same as in Embodiment 1.

[0088] As shown in Figure 22, when terminal 100 obtains prediction information of the object being watched from the central server 200 (S112), it performs matching (S114). The detection unit 120 detects objects from video input after the video in which the central server 200 performed behavior recognition, and the matching unit 133 matches the prediction information of the object being watched obtained from the central server 200 with the detection result of the object detected by the detection unit 120 from the input video.

[0089] In this embodiment, the prediction information of the object being watched predicted and notified by the central server 200 and the detection results of the object detected by the detection unit 120 include characteristic information such as the type of object, which is the object label, the position information of the rectangular area, and the image feature quantities of the object contained in the rectangular area.

[0090] Figure 23 shows an example of the matching process. In this example, matching is determined by comparing the object type, the image features of the object, and the object's position information, but matching may also be determined by comparing any one of these factors.

[0091] As shown in Figure 23, the matching unit 133 compares the type of object in the predicted information of the object being watched with the type of object in the object detection result (S301). The matching unit 133 determines whether the type of object included in the predicted information matches the type of object included in the detection result. The matching unit 133 determines that they match if the types of objects are the same or similar. Similar object types are the types of objects belonging to the same category, higher or lower category, and may be pre-set. For example, dump trucks and trucks are similar, so they may be determined to match.

[0092] Furthermore, the matching unit 133 compares the image features of the object in the prediction information of the object being watched with the image features of the object in the object detection result (S302). The matching unit 133 determines whether the image features within the object region included in the prediction information match the image features within the object region included in the detection result. For example, it compares image features such as HOG (Histograms of Oriented Gradients) or intermediate layer features of deep learning, and color features such as color histograms. The matching unit 133 determines whether they match based on the similarity of the image features. For example, it may determine that they match if the similarity is greater than a predetermined threshold.

[0093] Furthermore, the matching unit 133 compares the position information of the object in the predicted information of the object being watched with the position information of the object in the object detection result (S303). The comparison of position information includes comparing the position of the regions and comparing the size of the regions. The matching unit 133 determines whether the position information matches based on the distance between the object included in the predicted information and the object included in the detection result, the overlap between the rectangular region of the object included in the predicted information and the rectangular region of the object included in the detection result, and the difference between the size of the rectangular region of the object included in the predicted information and the size of the rectangular region of the object included in the detection result. The distance between the rectangular regions may be the distance between the centers of the rectangular regions, or the distance between any points included in the rectangular regions. The overlap of the rectangular regions is, for example, IoU. The size of the rectangular regions may be determined by calculating only the difference in size, regardless of position. For example, the matching unit 133 determines that the position information matches if the distance between the rectangular regions is smaller than a predetermined threshold, the overlap between the rectangular regions is larger than a predetermined threshold, or the difference in size between the rectangular regions is smaller than a predetermined threshold.

[0094] Next, the matching unit 133 decides whether or not to match based on these determination results (S304). For example, it may decide that the predicted information of the object being watched and the detection result of the object match if all comparison conditions—the type of object, the image characteristics of the object, and the location information—match. Alternatively, it may decide that to match if any of the comparison conditions—the type of object, the image characteristics of the object, or the location information—match, or if multiple arbitrarily selected comparison conditions match. For example, it may decide that to match if the type of object and the image characteristics of the object match, or if the type of object and the location information match, or if the image characteristics and the location information match.

[0095] Next, terminal 100 determines the gaze area based on the matching result (S115). The second determination unit 132 determines the gaze area based on the object detection result if, for example, the predicted information of the gaze target matches the object detection result. That is, the area indicated by the object detection result is set as the gaze area. If the predicted information of the gaze target does not match the object detection result, the gaze area may be determined based on the predicted information of the gaze target, or it may not be determined. If the gaze area is determined based on the predicted information of the gaze target, the area indicated by the predicted information of the gaze target is set as the gaze area. If the gaze area is not determined, high-quality encoding may not be performed. For example, the action recognition result score may be obtained from the center server 200, and if the predicted information of the gaze target does not match the object detection result, it may be determined whether or not to determine the gaze area based on the action recognition result score. If the score is smaller than a predetermined value, the gaze area is determined based on the predicted information, and if the score is larger than a predetermined value, the gaze area may not be determined. Furthermore, if object detection results cannot be obtained, a decision may be made on whether or not to determine the gaze area based on the action recognition result score.

[0096] In the example in Figure 24, the predicted information for the object being watched includes rectangular regions for the person and the hammer, and the detected object also includes rectangular regions for the person and the hammer. In this example, the rectangular regions for the person overlap, and the rectangular regions for the hammer overlap, so it is determined that the predicted information for the object being watched, which includes the person and the work object, matches with the detected object. In this case, the area of ​​the detected object, which includes the person and the work object, is designated as the area being watched. When the predicted information for the object being watched includes both the person and the work object, a match is determined for each of the person and the work object. If both the person and the work object match, the area including the person and the work object may be designated as the area being watched. Alternatively, if at least the person matches, the area including the person and the work object may be designated as the area being watched.

[0097] Furthermore, when predictive information for multiple objects of focus is obtained, the matching of the predictive information for each object with the object detection result is determined, and one of the regions is selected according to the matching result. Based on the selected region, the focus area is determined. For example, if the object detection result matches the predictive information for any of the objects of focus, the focus area may be determined based on the detection result of the matched object. If the object detection result does not match the predictive information for any of the objects of focus, the focus area may be determined based on the predictive information for the object that is closest to the object detection result.

[0098] Furthermore, if detection results for multiple objects are obtained, the matching of the predicted information of the object being watched with the detection results of each object is determined, and one of the regions is selected according to the matching result, and the area of ​​focus is determined based on the selected region. For example, if the detection result of any object matches the predicted information of the object being watched, the area of ​​focus may be determined based on the detection result of the matched object. If the detection results of multiple objects match, the area of ​​focus may be determined based on the detection result of the object that is closest to the predicted information of the object being watched. If none of the detection results of any object match the predicted information of the object being watched, the area of ​​focus may be determined based on the predicted information of the object being watched, or based on the detection result of the object that is closest to the predicted information of the object being watched.

[0099] As described above, in this embodiment, in addition to the configuration of Embodiment 1, information predicted from the action recognition results, etc., is matched with information detected from the actually acquired video, and the area to be enhanced in image quality and sharpened is determined based on the matching result. This ensures that the image quality of the area in the actually acquired video that matches the predicted object is guaranteed, thus reliably preventing errors in action recognition.

[0100] This disclosure is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. For example, in Embodiment 2, information predicted by the central server is matched with information detected by the terminal, but information obtained from behavior recognition may be matched with information detected by the terminal without the central server performing prediction. In other words, extracted information extracted by behavior recognition processing, such as behavior recognition results, may be fed back to the terminal from the central server. Furthermore, the processing flow described in the embodiments above is just one example, and the order of each process is not limited to the example above. The order of some processes may be changed, or some processes may be executed in parallel.

[0101] Each configuration in the above-described embodiment may consist of hardware, software, or both, and may consist of one piece of hardware or software, or multiple pieces of hardware or software. Each device and each function (process) may be realized by a computer 40 having a processor 41 such as a CPU (Central Processing Unit) and a memory 42 as a storage device, as shown in Figure 25. For example, a program for performing the method (image processing method) in the embodiment may be stored in the memory 42, and each function may be realized by executing the program stored in the memory 42 with the processor 41.

[0102] These programs, when loaded into a computer, include a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The programs may be stored on non-temporary computer-readable media or tangible storage media. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSDs), or other memory technologies, CD-ROMs, digital versatile discs (DVDs), Blu-ray® discs, or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices. The programs may be transmitted over temporary computer-readable media or communication media. Examples, but not limited to, include electrical, optical, acoustic, or other forms of propagating signals.

[0103] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure.

[0104] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) Image quality control means for controlling the image quality of the gaze region, including the object being gazed upon, in the input video; A recognition means that performs a recognition process to recognize the object being watched in the video of the gazed area whose image quality has been controlled, A prediction means predicts the position of the object being watched in video footage after the video footage in which the recognition process was performed, based on the extracted information extracted from the recognition process. Based on the predicted position of the object being watched, the image quality control means determines the area being watched in which the image quality is controlled in the subsequent video. A video processing system equipped with the following features. (Note 2) The extracted information includes the time-series location information of the object being observed. The video processing system described in Appendix 1. (Note 3) The time-series positional information of the object being watched includes the trajectory information of the object being watched obtained from the tracking process in the recognition process. The video processing system described in Appendix 2. (Note 4) The prediction means predicts the position of the object being watched based on the extended line obtained by extending the trajectory information. The video processing system described in Appendix 3. (Note 5) The extracted information includes the results of behavioral recognition for the object being watched. A video processing system as described in any one of the items 1 to 4 of the appendix. (Note 6) The prediction means predicts the position of the object being watched based on the object used in the action indicated by the action recognition result. The video processing system described in Appendix 5. (Note 7) The prediction means predicts the position of the object being watched based on the orientation of the object being used. The video processing system described in Appendix 6. (Note 8) The prediction means predicts the position of the object being watched based on the orientation of the person performing the action indicated by the action recognition result. A video processing system as described in any one of the items 5 to 7 of the appendix. (Note 9) The system includes detection means for detecting objects from video input after the video that has undergone the aforementioned recognition processing, The determination means determines the gaze area based on the matching result between the gaze target whose position has been predicted and the detected object. A video processing system as described in any one of the items 1 to 8 of the appendix. (Note 10) The determination means performs matching based on the type of object, image features, or location information of the gaze target whose position has been predicted and the detected object. The video processing system described in Appendix 9. (Note 11) The determination means determines that the object being watched whose position was predicted and the object being detected are a match if the type of the object being watched whose position was predicted and the type of the object being detected are the same or similar. The video processing system described in Appendix 10. (Note 12) The determination means determines that the object whose position was predicted matches the object whose position was predicted if the similarity between the features of the image including the object whose position was predicted and the features of the image including the detected object is greater than a predetermined value. The video processing system described in Appendix 10. (Note 13) The determination means determines that the object being watched and the detected object match if the distance between the object whose position is predicted and the detected object is less than a predetermined value, if the overlap between the area of ​​the object being watched and the area of ​​the detected object is greater than a predetermined value, or if the difference between the size of the area of ​​the object being watched and the size of the area of ​​the detected object is less than a predetermined value. The video processing system described in Appendix 10. (Note 14) If the determination means determines that the gaze target whose position has been predicted matches the detected object, it determines the gaze area based on the detected object. The video processing system described in any one of the appendices 9 to 13. (Note 15) If the determination means determines that the gaze target whose position has been predicted does not match the detected object, it determines the gaze area based on the gaze target whose position has been predicted, or does not determine the gaze area. A video processing system as described in any one of the items 9 to 14 of the appendix. (Note 16) The determination means selects a region from either the multiple gaze targets whose positions have been predicted or the detected object, according to the matching result between the multiple gaze targets whose positions have been predicted and the detected object, and determines the gaze region based on the selected region. A video processing system as described in any one of the items 9 to 15 of the appendix. (Note 17) The determination means selects a region from either the gaze target whose position has been predicted or one of the multiple detected objects, in accordance with the matching result between the gaze target whose position has been predicted and the multiple detected objects, and determines the gaze region based on the selected region. A video processing system as described in any one of the appendices 9 to 16. (Note 18) The determination means determines whether or not to determine the gaze area based on the recognition result in the recognition process. A video processing system as described in any one of the items 1 through 17 of the appendix. (Note 19) The determination means determines the gaze area if the score of the recognition result is less than a predetermined value. The video processing system described in Appendix 18. (Note 20) The object of attention includes the person who is the subject of the recognition process and the object used by the person, The area of ​​focus includes the area of ​​the person and the area of ​​the object being used. A video processing system as described in any one of the items 1 through 18 of the appendices. (Note 21) The image quality control means enhances the image quality of the gaze area compared to other areas. A video processing system as described in any one of the items 1 to 20 of the appendix. (Note 22) Controls the image quality of the gaze area, including the object being gazed upon, in the input video. A recognition process is performed to recognize the object being watched in the video from which the image quality of the gaze region has been controlled. Based on the extracted information obtained from the recognition process, the position of the object being watched in the video after the video in which the recognition process was performed is predicted. Based on the predicted position of the object being watched, the watching region in which the image quality is controlled in the subsequent video is determined. Image processing methods. (Note 23) The extracted information includes the time-series location information of the object being observed. The video processing method described in Appendix 22. (Note 24) The extracted information includes the results of behavioral recognition for the object being watched. The image processing method described in Appendix 22 or 23. (Note 25) Based on the object used in the action indicated by the action recognition result, the position of the object being watched is predicted. The video processing method described in Appendix 24. (Note 26) Based on the orientation of the person performing the action indicated by the action recognition result, the position of the object being watched is predicted. The image processing method described in Appendix 24 or 25. (Note 27) The system detects objects from video footage input after the video footage that has undergone the aforementioned recognition process. Based on the matching result between the gaze target whose position has been predicted and the detected object, the gaze area is determined. The image processing method described in any one of the items 22 to 26 of the appendix. (Note 28) The object of attention includes the person who is the subject of the recognition process and the object used by the person, The area of ​​focus includes the area of ​​the person and the area of ​​the object being used. The image processing method described in any one of the items 22 to 27 of the appendix. (Note 29) Image quality control means for controlling the image quality of the gaze region, including the object being gazed upon, in the input video; A recognition means that performs a recognition process to recognize the object being watched in the video of the gazed area whose image quality has been controlled, A prediction means predicts the position of the object being watched in video footage after the video footage in which the recognition process was performed, based on the extracted information extracted from the recognition process. Based on the predicted position of the object being watched, the image quality control means determines the area being watched in which the image quality is controlled in the subsequent video. A video processing device equipped with the following features. (Note 30) The extracted information includes the time-series location information of the object being observed. The image processing device described in Appendix 29. (Note 31) The extracted information includes the results of behavioral recognition for the object being watched. The image processing apparatus described in Appendix 29 or 30. (Note 32) The prediction means predicts the position of the object being watched based on the object used in the action indicated by the action recognition result. The video processing device described in Appendix 31. (Note 33) The prediction means predicts the position of the object being watched based on the orientation of the person performing the action indicated by the action recognition result. The image processing apparatus described in Appendix 31 or 32. (Note 34) The object of attention includes the person who is the subject of the recognition process and the object used by the person, The area of ​​focus includes the area of ​​the person and the area of ​​the object being used. The image processing apparatus described in any one of the items 29 to 33 of the appendix. (Note 35) Controls the image quality of the gaze area, including the object being gazed upon, in the input video. A recognition process is performed to recognize the object being watched in the video from which the image quality of the gaze region has been controlled. Based on the extracted information obtained from the recognition process, the position of the object being watched in the video after the video in which the recognition process was performed is predicted. Based on the predicted position of the object being watched, the watching region in which the image quality is controlled in the subsequent video is determined. A video processing program that instructs a computer to perform a particular task. [Explanation of Symbols]

[0105] 1. Remote monitoring system 10. Video Processing System 11 Image Quality Control Unit 12 Recognition part 13 Prediction Section 14. Decision-making section 20 Image Processing Equipment 40 Computers 41 processors 42 memory 100 devices 101 Camera 102 Compression Efficiency Optimization Function 103 Video transmission function 110 Video Acquisition Unit 120 Detection unit 130 Image quality change determination unit 131 First determination unit 132 Second determination section 133 Matching Department 140 Compression efficiency determination unit 150 Terminal Communications Unit 200 Center Servers 201 Video Recognition Function 202 Alert generation function 203 GUI drawing function 204 Screen display function 210 Center Communications Department 220 Decoders 230 Behavior Recognition Department 231 Object detection unit 232 Tracking Section 233a Relevance Analysis Department 233b Behavior Predictor 234 Behavior Judgment Department 240 Extracted information storage section 250 Target Analysis Unit 260 Target position prediction unit 300 base stations 400 MEC 401 Compression Bitrate Control Function 402 Terminal control function

Claims

1. Image quality control means for controlling the image quality of the gaze area, including the object of gaze, in the input video to be higher quality than other areas, A recognition means that performs a recognition process to recognize the object being watched in the video of the gazed area whose image quality has been controlled, A prediction means predicts the position of the object being watched in video footage after the video footage in which the recognition process was performed, based on the extracted information extracted from the recognition process. Based on the predicted position of the object being watched, the image quality control means determines the area being watched in which the image quality is controlled in the subsequent video. Equipped with, The extracted information includes the results of behavioral recognition for the object being watched, The prediction means predicts the position of the object being watched based on the object used in the action indicated by the action recognition result. Video processing system.

2. The extracted information includes the time-series location information of the object being observed. The image processing system according to claim 1.

3. The prediction means predicts the position of the object being watched based on the orientation of the person performing the action indicated by the action recognition result. The image processing system according to claim 1.

4. The system includes detection means for detecting objects from video input after the video that has undergone the aforementioned recognition processing, The determination means determines the gaze area based on the matching result between the gaze target whose position has been predicted and the detected object. The image processing system according to claim 1 or 2.

5. The object of attention includes the person who is the subject of the recognition process and the object used by the person, The area of ​​focus includes the area of ​​the person and the area of ​​the object being used. The image processing system according to claim 1 or 2.

6. The image quality of the gaze area, including the object being gazed upon, in the input video is controlled to be higher quality than other areas. A recognition process is performed to recognize the object being watched in the video from which the image quality of the gaze region has been controlled. Based on the extracted information obtained from the recognition process, the position of the object being watched in the video after the video in which the recognition process was performed is predicted. Based on the predicted position of the object being watched, the watching region in which the image quality is controlled in the subsequent video is determined. The extracted information includes the results of behavioral recognition for the object being watched, In the aforementioned prediction, the position of the object being watched is predicted based on the object used in the action indicated by the action recognition result. Image processing methods.

7. Image quality control means for controlling the image quality of the gaze area, including the object of gaze, in the input video to be higher quality than other areas, A recognition means that performs a recognition process to recognize the object being watched in the video of the gazed area whose image quality has been controlled, A prediction means predicts the position of the object being watched in video footage after the video footage in which the recognition process was performed, based on the extracted information extracted from the recognition process. Based on the predicted position of the object being watched, the image quality control means determines the area being watched in which the image quality is controlled in the subsequent video. Equipped with, The extracted information includes the results of behavioral recognition for the object being watched, The prediction means predicts the position of the object being watched based on the object used in the action indicated by the action recognition result. Image processing device.

8. The image quality of the gaze area, including the object being gazed upon, in the input video is controlled to be higher quality than other areas. A recognition process is performed to recognize the object being watched in the video from which the image quality of the gaze region has been controlled. Based on the extracted information obtained from the recognition process, the position of the object being watched in the video after the video in which the recognition process was performed is predicted. Based on the predicted position of the object being watched, the watching region in which the image quality is controlled in the subsequent video is determined. The extracted information includes the results of behavioral recognition for the object being watched, In the aforementioned prediction, the position of the object being watched is predicted based on the object used in the action indicated by the action recognition result. A program that causes a computer to perform a process.

Images