Image processing system, image processing method, image processing device, and image processing program

Abstract

A video processing system (10) comprises: an image quality control unit (11) that controls the image quality of an observation region including a target to be observed in a video input to the video processing system (10); a recognition unit (12) that performs recognition processing relating to an object included in the video for which the image quality of the observation region is controlled by the image quality control unit (11); and an extraction unit (13) that, on the basis of a temporal change in the recognition results of the recognition processing performed by the recognition unit (12), extracts the target to be observed included in the observation region controlled by the image quality control unit (11).

Description

Image processing system, image processing method, and image processing device

[0001] The present disclosure relates to a video processing system, a video processing method, and a video processing device.

[0002] Development is underway for systems that perform surveillance and other functions by applying machine learning-based detection and recognition technologies to video footage captured by cameras.

[0003] A related technique is known, for example, from Patent Document 1. Patent Document 1 describes a technique for a surveillance camera that transmits captured video via a network, in which a region to which a target object detected from the video is to move is predicted, and the amount of code for the predicted region is controlled so as to be large.

[0004] Japanese Patent Application Laid-Open No. 2021-150739

[0005] In related technologies such as that disclosed in Patent Document 1, the area predicted from the detected object is controlled so that the amount of code is always large, i.e., so that the image quality is always high. For this reason, it may be difficult to appropriately reduce the amount of video data in the related technologies.

[0006] In view of the above problems, an object of the present disclosure is to provide a video processing system, a video processing method, and a video processing device that are capable of appropriately controlling the amount of video data.

[0007] The image processing system according to the present disclosure comprises an image quality control means for controlling the image quality of a gaze area including a gaze target in an input image, a recognition means for performing recognition processing on an object included in the image in which the image quality of the gaze area has been controlled, and an extraction means for extracting the gaze target based on changes over time in the recognition results of the recognition processing.

[0008] The image processing method according to the present disclosure controls the image quality of a gaze area in an input image that includes a gaze target, performs recognition processing on an object included in the image in which the image quality of the gaze area has been controlled, and extracts the gaze target based on changes over time in the recognition results of the recognition processing.

[0009] The video processing device according to the present disclosure comprises an image quality control means for controlling the image quality of a gaze area including a gaze target in an input video, a recognition means for performing recognition processing on an object included in the video in which the image quality of the gaze area has been controlled, and an extraction means for extracting the gaze target based on changes over time in the recognition results of the recognition processing.

[0010] According to the present disclosure, it is possible to provide a video processing system, a video processing method, and a video processing device that are capable of appropriately controlling the amount of video data.

[0011] 1 is a configuration diagram showing an overview of a video processing system according to an embodiment. FIG. 2 is a configuration diagram showing an overview of a video processing device according to an embodiment. FIG. 3 is a flowchart showing an overview of a video processing method according to an embodiment. FIG. 4 is a configuration diagram showing a basic configuration of a remote monitoring system according to embodiment 1. FIG. 5 is a configuration diagram showing an example of a configuration of a terminal according to embodiment 1. FIG. 6 is a configuration diagram showing an example of a configuration of a center server according to embodiment 1. FIG. 7 is a configuration diagram showing an example of a configuration of a behavior recognition unit according to embodiment 1. FIG. 8 is a flowchart showing an example of an operation of the remote monitoring system according to embodiment 1. FIG. 9 is a diagram for explaining video acquisition processing according to embodiment 1. FIG. 10 is a diagram for explaining object detection processing according to embodiment 1. FIG. 11 is a flowchart showing an example of an operation of behavior recognition processing according to embodiment 1. FIG. 12 is a diagram for explaining behavior recognition processing according to embodiment 1. FIG. 13 is a flowchart showing an example of an operation of gaze target extraction processing according to embodiment 1. FIG. 14 is a graph for explaining gaze target extraction processing according to embodiment 1. FIG. 15 is a graph for explaining gaze target extraction processing according to embodiment 1. FIG. 16 is a diagram for explaining gaze area determination processing according to embodiment 1. FIG. 17 is a flowchart showing an example of an operation of gaze target extraction processing according to embodiment 2. FIG. 18 is a graph for explaining gaze target extraction processing according to embodiment 2. FIG. 19 is a configuration diagram showing an overview of hardware of a computer according to an embodiment.

[0012] Hereinafter, embodiments will be described with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and redundant description will be omitted as necessary.

[0013] In a system that collects video via a network and recognizes objects, actions, etc. in the video, it is preferable to minimize the amount of video data to be transmitted, since the bandwidth of the network that transmits the video is limited. For example, the amount of video data can be reduced by increasing the video compression rate. However, if the video compression rate is high or if the data loss rate is high, the number of false recognitions increases, resulting in a decrease in recognition accuracy. Therefore, in the embodiment, it is possible to prevent false recognition while minimizing the amount of video data to be transmitted. For example, false recognition of important events, such as unsafe or dangerous behavior, can be prevented.

[0014] (Outline of the embodiment) First, an outline of the embodiment will be described. Fig. 1 shows a schematic configuration of a video processing system 10 according to the embodiment. The video processing system 10 can be applied to, for example, a remote monitoring system that collects video via a network and monitors the video.

[0015] As shown in FIG. 1, the video processing system 10 includes an image quality control unit 11, a recognition unit 12, and an extraction unit 13.

[0016] The image quality control unit 11 controls the image quality of a gaze area including a gaze target in an input video. For example, the image quality control unit 11 may improve the image quality of the gaze area, i.e., make it clearer, than other areas. The recognition unit 12 performs recognition processing on an object included in the video in which the image quality of the gaze area has been controlled by the image quality control unit 11. For example, the object recognition processing is behavior recognition processing that recognizes the behavior of the object, but it may also be processing that recognizes information or features related to other objects.

[0017] The extraction unit 13 extracts a gaze target based on a temporal change in the recognition result of the recognition process by the recognition unit 12. For example, the recognition result includes a score of an action label of an action recognized by the action recognition process. The action label indicates the type of action of the recognized object, and the score of the action label indicates the probability that the object is performing the action of the action label. For example, the extraction unit 13 may extract a gaze target based on the temporal increase or temporal variability of the score of the action label. The temporal increase of the score is the amount by which the score increases over time, and the temporal variability of the score is the variability of the score over time. Extracting a gaze target means selecting a gaze target from candidates included in the recognition result.

[0018] The video processing system 10 may be configured with a single device or multiple devices. FIG. 2 shows the configuration of a video processing device 20 according to an embodiment. As shown in FIG. 2, the video processing device 20 may include the image quality control unit 11, the recognition unit 12, and the extraction unit 13 shown in FIG. 1. The image quality control unit 11, the recognition unit 12, and the extraction unit 13 may each be implemented in separate devices. Part or all of the video processing system 10 may be located on an edge or in the cloud. For example, in a system that monitors video captured on-site via a network, an edge is a device located at or near the site, and is also a device close to the terminal in the network hierarchy. For example, the image quality control unit 11 may be located on an edge terminal, and the recognition unit 12 and the extraction unit 13 may be located on a cloud server. Furthermore, each function may be distributed to the cloud.

[0019] Fig. 3 shows a video processing method according to an embodiment. For example, the video processing method according to the embodiment is executed by the video processing system 10 of Fig. 1 or the video processing device 20 of Fig. 2. As shown in Fig. 3, first, the image quality of a gaze area including a gaze target in an input video is controlled (S11). Next, a recognition process is performed on an object included in the video in which the image quality of the gaze area has been controlled (S12). Next, the gaze target is extracted based on a temporal change in the recognition result of the recognition process (S13). Then, returning to S11, the image quality of the gaze area including the extracted gaze target is controlled for the input video.

[0020] As described above, the video processing system according to the embodiment recognizes the behavior of an object from video whose image quality has been controlled, and extracts a target of attention based on temporal changes in the recognition results. For example, if the recognition results change significantly over time, it is more likely that the target is one that should be focused on. Therefore, by designating the object corresponding to the recognition result as the target of attention, it is possible to appropriately select the area whose image quality is to be controlled. Furthermore, since areas other than the target of attention can be compressed, it is possible to prevent erroneous recognition while reducing the amount of video data to be transmitted.

[0021] (First Embodiment) Next, a first embodiment will be described. First, a remote monitoring system, which is an example of a system to which this embodiment is applied, will be described. FIG. 4 shows the basic configuration of the remote monitoring system 1. The remote monitoring system 1 is a system that monitors an area captured by a camera using video. In this embodiment, the system will be described as a system that remotely monitors the work of workers at a site. For example, the site may be an area where people and machines are operating, such as a work site such as a construction site, a public square where people gather, or a school. In this embodiment, the work will be described as construction work, civil engineering work, etc., but is not limited to this. Note that video includes multiple images (also referred to as frames) in a time series, so the terms video and image are interchangeable. In other words, the remote monitoring system can be considered a video processing system that processes video and an image processing system that processes images.

[0022] As shown in FIG. 4 , the remote monitoring system 1 includes multiple terminals 100, a center server 200, a base station 300, and an MEC 400. The terminals 100, the base station 300, and the MEC 400 are located on the field side, and the center server 200 is located on the center side. For example, the center server 200 is located in a data center, a monitoring center, or the like located away from the field. The field side is the edge side of the system, and the center side is also the cloud side. The center server 200 may be configured as a single device or multiple devices. Furthermore, part or all of the center server 200 may be located in the cloud. For example, the image recognition function 201 and the alert generation function 202 may be located in the cloud, and the GUI drawing function 203 and the screen display function 204 may be located in a monitoring center, or the like.

[0023] The terminal 100 and the base station 300 are communicatively connected via a network NW1. The network NW1 is, for example, a wireless network such as 4G, local 5G / 5G, LTE (Long Term Evolution), or wireless LAN. The base station 300 and the center server 200 are communicatively connected via a network NW2. The network NW2 includes, for example, a core network such as 5GC (5th Generation Core network) or EPC (Evolved Packet Core), or the Internet. It can also be said that the terminal 100 and the center server 200 are communicatively connected via the base station 300. The base station 300 and the MEC 400 are communicatively connected via any communication method, but the base station 300 and the MEC 400 may be a single device.

[0024] The terminal 100 is a terminal device connected to the network NW1 and also serves as an image generating device that generates images of the site. The terminal 100 acquires images captured by a camera 101 installed at the site and transmits the acquired images to the center server 200 via the base station 300. The camera 101 may be located outside the terminal 100 or inside the terminal 100.

[0025] The terminal 100 compresses video from the camera 101 to a predetermined bit rate and transmits the compressed video. The terminal 100 has a compression efficiency optimization function 102 that optimizes compression efficiency, and a video transmission function 103. The compression efficiency optimization function 102 performs ROI control that controls the image quality of an ROI (Region of Interest; also called a fixation region). The compression efficiency optimization function 102 reduces the bit rate by maintaining the image quality of an ROI that includes a person or object while lowering the image quality of the surrounding area. The video transmission function 103 transmits the video with controlled image quality to the center server 200.

[0026] The base station 300 is a base station device of the network NW1, and also a relay device that relays communication between the terminal 100 and the center server 200. For example, the base station 300 is a local 5G base station, a 5G gNB (next generation Node B), an LTE eNB (evolved Node B), a wireless LAN access point, or the like, but may also be another relay device.

[0027] The MEC (Multi-access Edge Computing) 400 is an edge processing device located on the edge side of the system. The MEC 400 is an edge server that controls the terminal 100 and has a compression bit rate control function 401 and a terminal control function 402 that control the bit rate of the terminal. The compression bit rate control function 401 controls the bit rate of the terminal 100 through adaptive video distribution control and QoE (quality of experience) control. For example, the compression bit rate control function 401 predicts the recognition accuracy to be obtained while suppressing the bit rate according to the communication environment of the networks NW1 and NW2, and allocates a bit rate to the camera 101 of each terminal 100 so as to improve the recognition accuracy. The terminal control function 402 controls the terminal 100 to transmit video at the allocated bit rate. The terminal 100 encodes the video so as to obtain the allocated bit rate and transmits the encoded video.

[0028] The center server 200 is a server installed on the center side of the system. The center server 200 may be one or more physical servers, or may be a cloud server built on the cloud or other virtualized server. The center server 200 is a monitoring device that monitors on-site work by recognizing the work of people from on-site camera images. The center server 200 is also a recognition device that recognizes the behavior of people in the images transmitted from the terminal 100.

[0029] The center server 200 has an image recognition function 201, an alert generation function 202, a GUI drawing function 203, and a screen display function 204. The image recognition function 201 inputs the image transmitted from the terminal 100 into an image recognition AI (Artificial Intelligence) engine to recognize the work performed by the worker, i.e., the type of human behavior. The alert generation function 202 generates an alert according to the recognized work. The GUI drawing function 203 displays a GUI (Graphical User Interface) on the screen of the display device. The screen display function 204 displays the image of the terminal 100, the recognition results, alerts, etc. on the GUI.

[0030] Next, a description will be given of a specific example of the remote monitoring system 1. In this specific example of the present embodiment, a gaze target is extracted based on a temporal change in the behavior recognition result.

[0031] Fig. 5 shows an example of the configuration of the terminal 100 according to this embodiment, and Fig. 6 shows an example of the configuration of the center server 200 according to this embodiment. Note that the configuration of each device is merely an example, and other configurations may be used as long as the operations according to this embodiment, which will be described later, are possible. For example, some of the functions of the terminal 100 may be arranged in the center server 200 or another device, or some of the functions of the center server 200 may be arranged in the terminal 100 or another device.

[0032] As shown in FIG. 5, the terminal 100 includes a video acquisition unit 110 , a detection unit 120 , an image quality change determination unit 130 , a compression efficiency determination unit 140 , and a terminal communication unit 150 .

[0033] The video acquisition unit 110 acquires video (also referred to as input video) captured by the camera 101. For example, the input video may include a person who is a worker performing work on-site, a work object used by the person, etc. The video acquisition unit 110 also serves as an image acquisition unit that acquires multiple images in time series.

[0034] The detection unit 120 is an object detection unit that detects objects in the acquired input video. The detection unit 120 detects objects in each image included in the input video and assigns labels to the detected objects, i.e., object labels. An object label is an object class and indicates the type of object. For example, object labels include person, car, robot, hammer, etc. The detection unit 120 extracts a rectangular area containing an object from each image included in the input video, recognizes the object within the extracted rectangular area, and assigns a label to the recognized object. The rectangular area is a bounding box or an object area. Note that the object area containing the object is not limited to a rectangular area and may be a circular or irregular silhouette area, etc. The detection unit 120 calculates image features of the object contained in the rectangular area and recognizes the object based on the calculated features. For example, the detection unit 120 recognizes objects in the image using an object recognition engine that uses machine learning such as deep learning. Objects can be recognized by machine learning the image features and object labels of the object. The object detection result includes the object label, position information of the rectangular area containing the object, etc. The object position information is, for example, the coordinates of each vertex of a rectangular area, but may also be the position of the center of the rectangular area or the position of any point on the object. The detection unit 120 transmits the object detection result to the image quality change determination unit 130.

[0035] The image quality change determination unit 130 determines a region of interest (ROI), which is an image quality change region in which the image quality of the acquired input video is changed. The image quality change determination unit 130 is a determination unit that determines the region of interest. The region of interest is a region that includes an object of attention, and is a region in which the image quality is improved, i.e., sharpened. The region of interest can also be said to be a region that ensures image quality for behavior recognition.

[0036] For example, the image quality change determination unit 130 includes a first determination unit 131 and a second determination unit 132. For example, the first determination unit 131 first determines a gaze area, and after the center server 200 recognizes the behavior, the second determination unit 132 determines the gaze area. Note that the determination of the gaze area by the first determination unit 131 may be omitted, and only the determination of the gaze area by the second determination unit 132 may be performed. The first determination unit 131 determines the gaze area of ​​the input video based on the detection result of an object detected in the input video by the detection unit 120. The first determination unit 131 determines the gaze area based on position information of an object that has a label indicating the gaze target among the detected objects detected in the input video by the detection unit 120. The gaze target may be a person who is the target of behavior recognition, or may include a work object that the person can use in their work. For example, the label of the work object is preset as the label of an object associated with the person. Note that the target of behavior recognition is not limited to a person, but may also be an object such as heavy machinery or a robot. That is, actions including work performed by heavy machinery or robots may be recognized.

[0037] When information is fed back from the center server 200 that recognized the behavior, the second determination unit 132 determines the gaze area of ​​the input video based on the fed back information. In this example, extracted gaze target information, which is information on the gaze target extracted by the center server 200, is fed back. The extracted gaze target information is information on the gaze target, and is information indicating the gaze target extracted by the center server 200 through behavior recognition. The extracted gaze target information is position information of the gaze target, and includes position information of a rectangular area of ​​the gaze target. For example, the second determination unit 132 determines the rectangular area indicated by the acquired extracted gaze target information as the gaze area. In other words, the second determination unit 132 determines an area that ensures the image quality of the input video based on the position of the extracted gaze target.

[0038] The compression efficiency determination unit 140 determines a compression rate for the attention area or an area other than the attention area and compresses the video. The compression efficiency determination unit 140 is an encoder that encodes the input video using the determined compression rate. The compression efficiency determination unit 140 encodes the input video using a video encoding method such as H.264 or H.265. The compression efficiency determination unit 140 also encodes the input video so that the bit rate is equal to the bit rate assigned by the compression bit rate control function 401 of the MEC 400.

[0039] The compression efficiency determination unit 140 is an image quality control unit that controls the image quality of the attention area determined by the image quality change determination unit 130 and corresponds to the image quality control unit 11 in FIG. 1 . The compression efficiency determination unit 140 can also be considered an image quality improvement unit that improves the image quality of the attention area. The attention area is an area determined by either the first determination unit 131 or the second determination unit 132. The compression efficiency determination unit 140 compresses the attention area and other areas at predetermined compression rates, respectively, to encode the image quality of the attention area to a predetermined quality. In other words, by changing the compression rates of the attention area and other areas, the image quality of the attention area is improved compared to other areas. It can also be said that the image quality of other areas is reduced compared to the attention area. For example, the image quality of the attention area and other areas is controlled within the bit rate range assigned by the compression bit rate control function 401 of the MEC 400. The image quality of the attention area may be controlled not only by the compression rate but also by changing the image resolution, frame rate, etc. The image quality of the gaze area may also be controlled by changing the amount of color information in the image, for example, color, grayscale, black and white, etc.

[0040] The terminal communication unit 150 transmits the encoded data encoded by the compression efficiency determination unit 140 to the center server 200 via the base station 300. The terminal communication unit 150 is a transmission unit that transmits video in which the image quality of the gaze area has been controlled. The terminal communication unit 150 also receives extracted gaze target information transmitted from the center server 200 via the base station 300. The terminal communication unit 150 is an acquisition unit that acquires the extracted gaze target information. The terminal communication unit 150 is an interface capable of communicating with the base station 300, and is, for example, a wireless interface such as 4G, local 5G / 5G, LTE, or wireless LAN, but may also be a wireless or wired interface of any other communication method. The terminal communication unit 150 may include a first terminal communication unit that transmits the encoded data and a second terminal communication unit that receives the extracted gaze target information. The first terminal communication unit and the second terminal communication unit may be communication units of the same communication method or communication units of different communication methods.

[0041] As shown in FIG. 6 , the center server 200 includes a center communication unit 210 , a decoder 220 , a behavior recognition unit 230 , an analysis information storage unit 240 , and a gaze target analysis unit 250 .

[0042] The center communication unit 210 receives encoded data transmitted from the terminal 100 via the base station 300. The center communication unit 210 is a receiving unit that receives video in which the image quality of the gaze area has been controlled. The center communication unit 210 also transmits extracted gaze target information extracted by the gaze target analysis unit 250 to the terminal 100 via the base station 300. The center communication unit 210 is a notifying unit that notifies the extracted gaze target information. The center communication unit 210 is an interface capable of communicating with the Internet or a core network, and is, for example, a wired interface for IP communication, but may also be a wired or wireless interface of any other communication method. The center communication unit 210 may include a first center communication unit that receives encoded data and a second center communication unit that transmits extracted gaze target information. The first center communication unit and the second center communication unit may be communication units using the same communication method or different communication methods.

[0043] The decoder 220 decodes the encoded data received from the terminal 100. The decoder 220 decodes using a video encoding method compatible with the encoding method of the terminal 100, such as H.264 or H.265. The decoder 220 performs decoding according to the compression rate of each region and generates decoded video (also referred to as received video).

[0044] The behavior recognition unit 230 is a recognition unit that recognizes the behavior of an object in the decoded received video, and corresponds to the recognition unit 12 in FIG. 1. The behavior recognition unit 230 executes behavior recognition processing to recognize the behavior of the gaze target for the video in which the image quality of the gaze area has been controlled. The behavior recognition unit 230 detects an object from the received video and recognizes the behavior of the detected object. The behavior recognition unit 230 recognizes the behavior of a person who is the target of behavior recognition and assigns a label to the recognized behavior, i.e., a behavior label. The behavior label is a class of behavior and indicates the type of behavior.

[0045] For example, the behavior recognition unit 230 recognizes a person's behavior based on the person and work object detected from the received video. The behavior recognition unit 230 may recognize a person's behavior by identifying the relationship between the person and the work object. The relationship between the person and the work object includes which object the person is using or not using. For example, a work object for each person may be identified based on the distance between the person and the work object, and behavior may be recognized from the identified work object. For example, the behavior recognition unit 230 performs machine learning on the work object and task associated with the person, and recognizes the person's behavior based on machine learning. Note that, without being limited to machine learning, the behavior recognition unit 230 may associate the work object and task associated with the person and recognize the person's behavior based on rules. For example, the behavior recognition unit 230 may associate the work object and task associated with the person in advance, and recognize the person's behavior based on the detected work object. Alternatively, if a work object is not detected, behavior may be recognized from the person alone. For example, the behavior recognition unit 230 may associate the person's posture and shape with the task in advance, and recognize the person's behavior based on the detected posture and shape.

[0046] The analysis information storage unit 240 stores the analysis information analyzed by the behavior recognition unit 230. The analysis information includes behavior recognition results, person detection information, and work object detection information related to the behavior. Note that the behavior recognition results may include person detection information and work object detection information related to the behavior. The behavior recognition results include a label of the recognized behavior, a score of the behavior label, identification information of the person performing the recognized behavior, and identification information of the work object used in the recognized behavior. The behavior label score indicates the confidence level, which is the likelihood (probability) of the behavior label. The higher the score, the more likely the predicted behavior of the behavior label is to be correct. The person detection information includes position information of the person's rectangular area, tracking information, and the like. The tracking information is trajectory information indicating the tracking results of the object. The work object detection information includes an object label, an object label score, position information of the object's rectangular area, tracking information, and the like. For example, the behavior predictor (behavior recognition engine) of the behavior recognition unit 230 learns to weight objects related to the behavior, thereby extracting candidate work objects that may be related to each image and outputting information about the extracted candidate work objects. For example, if a pile driving operation is recognized, information about a hammer, which is an object related to the action, is output.

[0047] The gaze target analysis unit 250 is an extraction unit that extracts a gaze target based on the analysis information analyzed by the behavior recognition unit 230, and corresponds to the extraction unit 13 in FIG. 1 . The analysis information may be acquired from the behavior recognition unit 230 or from the analysis information storage unit 240. The gaze target analysis unit 250 determines a gaze target that ensures image quality to prevent behavior recognition errors based on the analyzed information. For example, the gaze target analysis unit 250 determines a gaze target based on the behavior recognition result. The gaze target analysis unit 250 determines a person whose behavior is recognized by the behavior recognition unit 230, i.e., a person whose behavior is included in the behavior recognition result, as the gaze target. When an action is recognized from a work object associated with a person, the person and the work object may be determined as the gaze target. There may be multiple work objects associated with a person, and the person and multiple work objects may be determined as the gaze targets. For example, when a pile driving operation is recognized, the objects related to the operation may be a "pile" and a "hammer," and the person, the "pile," and the "hammer" may be determined as the gaze targets.

[0048] In this embodiment, the gaze target analysis unit 250 extracts the gaze target based on a temporal change in the behavior recognition result. For example, if it is determined that the score of a behavior label of interest is increasing, i.e., increasing, based on the amount of increase in the score over time, the gaze target may be determined to be an object including a person who performed the behavior of the behavior label and a work object. Alternatively, the gaze target may be determined based on a comparison result between the behavior label score and a threshold, or based on the temporal variation in the behavior label score.

[0049] The gaze target analysis unit 250 outputs position information of the extracted gaze target rectangular area as extracted gaze target information. The position information is, for example, the coordinates of each vertex of the rectangular area, but may also be the center position of the rectangular area or the position of any point of the gaze target. The extracted gaze target information is not limited to position information as information about the extracted gaze target, and may also include information analyzed by the behavior recognition unit 230, such as the object label of the gaze target, image features, behavior label, and behavior label score.

[0050] Fig. 7 shows an example configuration of the behavior recognition unit 230 in the center server 200. Fig. 7 shows an example configuration in which behavior recognition based on the association between a person and a work object is performed based on machine learning. In the example of Fig. 7, the behavior recognition unit 230 includes an object detection unit 231, a tracking unit 232, a behavior predictor 233, and a behavior determination unit 234.

[0051] The object detection unit 231 detects objects in the input received video. For example, like the detection unit 120 of the terminal 100, the object detection unit 231 is a detection unit such as an object recognition engine using machine learning. That is, the object detection unit 231 extracts a rectangular area including an object from each image of the received video, recognizes the object in the extracted rectangular area, and assigns a label to the recognized object. The object detection result includes the object label and position information of the rectangular area including the object.

[0052] The tracking unit 232 tracks the detected object in the received video. Based on the object detection result, the tracking unit 232 associates the object in each image included in the received video. By assigning a tracking ID to the detected object, each object can be identified and tracked. For example, the object is tracked by associating the object between images based on the distance or overlap (e.g., IoU: Intersection over Union) between the rectangular area of ​​the object detected in the previous image and the rectangular area of ​​the object detected in the next image.

[0053] The behavior predictor 233 predicts the behavior of each object tracked by the tracking unit 232. The behavior predictor 233 recognizes the behavior of the person tracked in the received video and assigns a label to the recognized behavior. For example, the behavior predictor 233 recognizes the behavior of the person in the received video using a behavior recognition engine that uses machine learning such as deep learning. The behavior of the person can be recognized by machine learning the video of the person performing work using the work object and the behavior label. For example, machine learning is performed using learning data that is a video of the person performing work using the work object, annotation information such as the positions of the person and the work object and information related to the person and the object, and behavior information such as the work object required for each work. The behavior predictor 233 also outputs a score for the recognized behavior label.

[0054] The behavior determination unit 234 determines the behavior of the object based on the predicted behavior label. The behavior determination unit 234 determines the behavior of the person based on the score of the behavior label predicted by the behavior predictor 233. For example, the behavior determination unit 234 outputs the behavior label with the highest score as the recognition result.

[0055] Next, the operation of the remote monitoring system according to this embodiment will be described. Fig. 8 shows an example of the operation of the remote monitoring system 1. For example, the explanation will be given assuming that the terminal 100 executes S101 to S105 and S111 to S112, and the center server 200 executes S106 to S110, but this is not limiting and any device may execute each process.

[0056] As shown in Fig. 8 , the terminal 100 acquires video from the camera 101 (S101). The camera 101 generates video of a work site, and the video acquisition unit 110 acquires the video (input video) output from the camera 101. For example, as shown in Fig. 9 , the image of the input video includes a person working on the work site and a work object such as a hammer used by the person.

[0057] Next, the terminal 100 detects objects based on the acquired input video (S102). The detection unit 120 uses an object recognition engine to detect rectangular areas within an image included in the input video, recognize objects within the detected rectangular areas, and assign labels to the recognized objects. For each detected object, the detection unit 120 outputs the object label and position information of the object's rectangular area as the object detection result. For example, when object detection is performed from the image of FIG. 9, a person and a hammer are detected, and the rectangular area of ​​the person and the rectangular area of ​​the hammer are detected, as shown in FIG. 10.

[0058] Next, the terminal 100 determines a gaze area in the input video based on the object detection result (S103). The first determination unit 131 of the image quality change determination unit 130 extracts objects having a label that is the gaze target based on the object detection result for each object. The first determination unit 131 extracts objects whose object label is person or work object from the detected objects, and determines the rectangular area of ​​the corresponding object as the gaze area. In the example of Figure 10, a person and a hammer are detected in the image, and the hammer corresponds to the work object, so the rectangular area of ​​the person and the rectangular area of ​​the hammer are determined as the gaze area.

[0059] Next, the terminal 100 encodes the input video based on the determined gaze area (S104). The compression efficiency determination unit 140 encodes the input video so that the gaze area has higher image quality than other areas. In the example of Fig. 10, the compression rates of the rectangular area of ​​the person and the rectangular area of ​​the hammer are set lower than the compression rates of other areas, thereby improving the image quality of the rectangular area of ​​the person and the rectangular area of ​​the hammer.

[0060] Next, the terminal 100 transmits the encoded data to the center server 200 (S105), and the center server 200 receives the encoded data (S106). The terminal communication unit 150 transmits the encoded data with the image quality of the gaze area enhanced to the base station 300. The base station 300 transfers the received encoded data to the center server 200 via the core network or the Internet. The center communication unit 210 receives the transferred encoded data from the base station 300.

[0061] Next, the center server 200 decodes the received encoded data (S107). The decoder 220 decodes the encoded data in accordance with the compression rate of each region, and generates a video (received video) in which the image quality of the attention region is improved.

[0062] Next, the center server 200 recognizes the behavior of the object based on the decoded received video (S108). Fig. 11 shows an example of the behavior recognition process by the behavior recognition unit 230 shown in Fig. 7.

[0063] 11 , first, the object detection unit 231 detects objects in the input received video (S201). Using an object recognition engine, the object detection unit 231 detects rectangular areas in each image included in the received video, recognizes objects in the detected rectangular areas, and assigns labels to the recognized objects. For each detected object, the object detection unit 231 outputs the object label and position information of the object's rectangular area as the object detection result.

[0064] Next, the tracking unit 232 tracks the detected objects in the received video (S202). The tracking unit 232 assigns a tracking ID to each detected object, and tracks the object identified by the tracking ID in each image.

[0065] Next, the behavior predictor 233 predicts the behavior of each tracked object (S203). The behavior predictor 233 uses a behavior recognition engine to predict the behavior of the person from the video including the tracked person and the work object. The behavior predictor 233 outputs the label of the predicted behavior and the score of each behavior label.

[0066] Next, the behavior determination unit 234 determines the behavior of the object based on the scores of the predicted behavior labels (S204). In the example of FIG. 12, a person and a hammer are detected by tracking. The behavior predictor 233 recognizes the person's behavior based on the video of the detected person and hammer, and outputs a score for each behavior label. For example, the score for pile driving is 0.8, the score for heavy machinery operation is 0.1, the score for unsafe behavior is 0.0, and the score for non-work is 0.1. Then, the behavior determination unit 234 determines that the person's behavior is pile driving because the score for pile driving is the highest. The behavior determination unit 234 outputs the determined behavior and the behavior score.

[0067] Returning to FIG. 8 , following the behavior recognition process, the center server 200 extracts a gaze target based on the analysis information analyzed by the behavior recognition process (S109). The gaze target analysis unit 250 sets the person whose behavior has been recognized as the gaze target, and if the recognized target includes a work object, the work object may also be included as the gaze target. For example, in the example of FIG. 12 , since the work of driving a pile is recognized from the person and the hammer, the person whose work has been recognized and the hammer may be set as the gaze target. The gaze target analysis unit 250 outputs gaze target extraction information including position information of the extracted gaze target.

[0068] In this embodiment, a gaze target is extracted based on temporal changes in behavior recognition results. FIG. 13 shows an example of the operation of the gaze target extraction process according to this embodiment. When the behaviors of multiple objects are recognized, the gaze target extraction process is performed on the recognition results of each object. In the example of FIG. 13 , first, the gaze target analysis unit 250 extracts the score of a behavior label of interest (S301). The behavior label of interest is the behavior label of a behavior that the user wants to focus on among the behavior labels of recognizable behaviors. For example, a behavior label of interest is set in advance, and the gaze target analysis unit 250 extracts the score of the set behavior label from the behavior recognition results. For example, the score of the corresponding behavior label is extracted from the scores of multiple behavior labels generated by the behavior predictor 233. When there are multiple behavior labels of interest, the scores of the corresponding multiple behavior labels of interest may be extracted from the behavior recognition results. Note that the score of any behavior label, for example, the highest score, may be extracted from the scores of multiple behavior labels included in the behavior recognition results, not limited to the behavior label of interest.

[0069] Next, the gaze target analysis unit 250 determines the temporal change in the score of the activity label of interest (S302). For example, the gaze target analysis unit 250 determines whether the score is increasing, i.e., whether there is an upward trend, based on the temporal change in the score of the activity label of interest. For example, as shown in FIG. 14, it may determine whether the amount of change in the score of the activity label over a predetermined period, i.e., whether the rate of increase, is greater than a predetermined threshold. Furthermore, as shown in FIG. 15, it may determine whether the score of the activity label at multiple times (e.g., T0 to T3) is continuously increasing compared to the previous time.

[0070] The attention target analysis unit 250 may also determine whether the score of the behavior label of interest is greater than a predetermined threshold. For example, the scores of the behavior label at multiple times are compared with the threshold. The threshold may be set for each behavior label. For example, the attention target analysis unit 250 may determine whether the score exceeds the threshold consecutively at each time, or may determine the number of times the score exceeds the threshold at each time.

[0071] The gaze target analysis unit 250 may also determine whether the variation in the scores of the activity label of interest is large, for example, by determining whether the variation in the scores of the activity label over multiple time periods is within a predetermined range.

[0072] If it is determined that the score of the behavior label of interest is increasing over time, the gaze target analysis unit 250 determines the target object performing the behavior of the behavior label of interest as the gaze target (S303). In the example of FIG. 14, if the score increases three times in one minute of a predetermined period, it is determined that the score is on an upward trend, and the target object of the corresponding behavior label is determined as the gaze target. Also, in the example of FIG. 15, if the score continues to increase continuously at three points / s from T1 to T3, it is determined that the score is on an upward trend, and the target object of the corresponding behavior label is determined as the gaze target. If the score of the behavior label is on an upward trend, it is more likely that the behavior is one that requires attention, so the object of the behavior label is determined as the gaze target, and the behavior of the corresponding object can be accurately recognized by the high-resolution video.

[0073] Furthermore, when it is determined that the score of the behavior label of interest is greater than a threshold, the gaze target analysis unit 250 may determine the target object of the behavior label of interest as the gaze target. For example, when the score is greater than the threshold consecutively at 3 points / s, it is determined that the high score state continues, and the target object of the corresponding behavior label is determined as the gaze target. When the high score state of the behavior label continues, it is highly likely that the behavior to be focused on is continuing, so the object of the behavior label is determined as the gaze target, and the behavior of the corresponding object can be accurately recognized by the high-resolution video.

[0074] Furthermore, if it is determined that the variance in the scores of the activity label of interest is large, the gaze target analysis unit 250 may determine the target object of the activity label of interest as the gaze target. For example, if the score is 0.1 at time T = 1, 0.8 at time T = 2, 0.2 at time T = 3, and 0.9 at time T = 4, the score changes significantly over time. Therefore, it is determined that the variance is large, and the target object of the corresponding activity label is determined as the gaze target. Furthermore, the variance may be measured using the variance of the scores of the activity label in multiple past recognition results. If the variance exceeds a predefined threshold, it may be determined that the variance is large, and the target object of the corresponding activity label may be determined as the gaze target. If the scores of the activity labels vary, it is highly likely that the activity labels have not been correctly determined. Therefore, the object of the corresponding activity label is determined as the gaze target, enabling the activity of the corresponding object to be correctly determined using high-resolution video.

[0075] On the other hand, if it is determined that the score of the behavior label of interest has not increased over time, the gaze target analysis unit 250 excludes the target object performing the behavior of the behavior label of interest from the gaze targets (S304). That is, in this case, the gaze target analysis unit 250 does not select the object as a gaze target. In the example of FIG. 14 , if the score increase is less than double within one minute of the predetermined period, it is determined that the score is not on an upward trend, and the target object of the corresponding behavior label is excluded from the gaze targets. Furthermore, in the example of FIG. 15 , if the score has decreased by even one point out of the three points / s T1 to T3, it is determined that the score is not on an upward trend, and the target object of the corresponding behavior label is excluded from the gaze targets.

[0076] Furthermore, if it is determined that the score of the behavior label of interest is smaller than the threshold, the gaze target analysis unit 250 may exclude the target object of the behavior label of interest from the gaze targets. For example, if the score is lower than the threshold by even one point out of 3 points / s, it is determined that the high score state has not continued, and the target object of the corresponding behavior label is excluded from the gaze targets.

[0077] Furthermore, if it is determined that the variance in the score of the activity label of interest is small, the gaze target analysis unit 250 may exclude the target object of the activity label of interest from the gaze targets. For example, if the score is 0.7 at time T=1, 0.8 at time T=2, 0.8 at time T=3, and 0.9 at time T=4, the change in score over time is small, so it is determined that the variance is small, and the target object of the corresponding activity label is excluded from the gaze targets. Furthermore, the variance may be measured using the variance of the scores of the activity label in multiple past recognition results. If the variance is smaller than a predefined threshold, it may be determined that the variance is small, and the target object of the corresponding activity label may be excluded from the gaze targets.

[0078] 8 , following the gaze target extraction process, the center server 200 notifies the terminal 100 of the extracted gaze target information extracted by the gaze target extraction process (S110), and the terminal 100 acquires the extracted gaze target information (S111). The center communication unit 210 transmits the extracted gaze target information indicating the position of the extracted gaze target to the base station 300 via the Internet or core network. The base station 300 transfers the received extracted gaze target information to the terminal 100. The terminal communication unit 150 receives the transferred extracted gaze target information from the base station 300.

[0079] Next, the terminal 100 determines a gaze area based on the received extracted gaze target information (S112). The second determination unit 132 of the image quality change determination unit 130 determines the area indicated by the extracted gaze target information notified by the center server 200 as the gaze area. In the example of FIG. 16, the extracted gaze target information indicates a rectangular area of ​​the person and a rectangular area of ​​the hammer, and these areas are determined as the gaze area. Alternatively, a circumscribing area including the rectangular area of ​​the person and the rectangular area of ​​the hammer may be determined as the gaze area. This circumscribing area may be notified to the terminal 100 from the center server 200. Thereafter, S104 to S112 are repeated.

[0080] As described above, in this embodiment, in a system that recognizes the behavior of an object from video, a gaze target is extracted based on temporal changes in the behavior recognition result, and the image quality of the area including the extracted gaze target is sharpened. For example, if the behavior recognition result increases over time or varies over time, it is assumed that there is a high need to recognize the behavior of that object. Therefore, by making a judgment based on temporal changes in the behavior recognition result, it is possible to appropriately select the target to gaze on. Therefore, depending on the behavior recognition result, it is possible to ensure the image quality of a specific part including the target to be gazed on and compress other areas, thereby reducing the amount of data to be transmitted and preventing behavior recognition errors.

[0081] (Embodiment 2) Next, embodiment 2 will be described. In this embodiment, an example will be described in which an extraction target is extracted based on the positional relationship of objects included in the behavior recognition result. The configuration of the remote monitoring system according to this embodiment is the same as that of embodiment 1 shown in FIGS. 5 and 6, and therefore a description thereof will be omitted. Note that this embodiment can be implemented in combination with embodiment 1, and each configuration shown in embodiment 1 may be used as appropriate.

[0082] In this embodiment, the overall operation flow of the remote monitoring system is the same as that of FIG. 8 in Embodiment 1, and only the gaze target extraction process (S109) differs from that of Embodiment 1. FIG. 17 shows an example of the operation of the gaze target extraction process according to this embodiment. Note that the gaze target may be extracted by either the gaze target extraction process of Embodiment 1 or the gaze target extraction process of this embodiment, or the gaze target may be extracted by both the gaze target extraction process of Embodiment 1 and the gaze target extraction process of this embodiment.

[0083] 17 , first, the gaze target analysis unit 250 selects related objects and peripheral objects related to the behavior to be determined (S311). The related objects related to the behavior to be determined are related objects related to the behavior that the user wants to determine among recognizable behaviors. The related objects are work objects that a person uses in their work. For example, object labels of related objects related to the behavior to be determined are set in advance, and when the object label of an object included in the behavior recognition result corresponds to the set object label, the gaze target analysis unit 250 selects the object included in the behavior recognition result as the related object.

[0084] Furthermore, the gaze target analysis unit 250 selects peripheral objects around the related object in the video in which behavior recognition was performed. All objects other than the related object detected in the video by object detection may be selected as peripheral objects, or objects detected within a predetermined range around the related object in the image may be selected as peripheral objects. The peripheral objects may be objects with a specific object label or any object. For example, object labels for objects to be designated as peripheral objects may be set in advance, and objects with the set object labels may be selected as peripheral objects. In the example of FIG. 18 , when it is desired to determine the approach of heavy equipment as an unsafe behavior, a heavy equipment M1 associated with the approach of heavy equipment is selected as a related object from the behavior recognition results, and people P1 and P2 around the heavy equipment M1 are selected as peripheral objects.

[0085] Next, the gaze target analysis unit 250 calculates the positional relationship between the selected related object and the peripheral objects (S312). The gaze target analysis unit 250 calculates a proximity index on the image as the positional relationship between the related object and the peripheral objects. The proximity index on the image may be, for example, the Euclidean distance between the centers of the rectangular regions of the objects, the average Euclidean distance between the corners of the rectangular regions of the objects, or the degree of overlap (IoU) of the object rectangles. If multiple peripheral objects are selected, a proximity index between the related object and each peripheral object is calculated. In the example of FIG. 18 , the Euclidean distance between the center of the rectangular region of the heavy equipment M1 and the center of the rectangular region of the person P1 is calculated, and the Euclidean distance between the center of the rectangular region of the heavy equipment M1 and the center of the rectangular region of the person P2 is calculated.

[0086] Next, the gaze target analysis unit 250 determines whether the related object and the peripheral object are close to each other based on the calculated positional relationship (S313). If it is determined that the related object and the peripheral object are close to each other, the peripheral object is determined to be the gaze target (S314). For example, the peripheral object is determined to be the gaze target when the proximity index meets a certain threshold or when the temporal change in the proximity index exceeds a certain threshold. The peripheral object may be determined to be the gaze target when the Euclidean distance between the objects is smaller than a threshold or when the degree of overlap between the objects is greater than a threshold. In the example of FIG. 18, if the Euclidean distance between the center of the rectangular area of ​​the heavy equipment M1 and the center of the rectangular area of ​​the person P1 is smaller than a threshold, the person P1 is determined to be the gaze target. In the example of FIG. 19, after time Ta, the degree of overlap between the rectangular areas exceeds a threshold, so the peripheral object is determined to be the gaze target. Note that the gaze target may also be determined according to the temporal change in the positional relationship between the related object and the peripheral object. For example, when the distance between the related object and the peripheral object gradually decreases or the distance becomes closer than a threshold, the peripheral object may be determined as the gaze target. When the overlap between the area of ​​the related object and the area of ​​the peripheral object gradually increases or the overlap becomes greater than a threshold, the peripheral object may be determined as the gaze target. Furthermore, the gaze target may be determined based on the orientations of the related object and the peripheral object, not limited to the positional relationship between the related object and the peripheral object. For example, when the orientation of a person, which is a peripheral object, is facing the related object, the peripheral object may be determined as the gaze target.

[0087] Furthermore, if it is determined that the related object and the peripheral object are far apart, the gaze target analysis unit 250 excludes the peripheral object from the gaze target (S315). That is, the peripheral object is not selected as the gaze target. For example, if the proximity index does not satisfy a certain threshold or if the temporal change in the proximity index does not exceed a certain threshold, the peripheral object is excluded from the gaze target. If the Euclidean distance between the objects is greater than a threshold or if the degree of overlap between the objects is less than a threshold, the peripheral object may be excluded from the gaze target. In the example of FIG. 18 , if the Euclidean distance between the center of the rectangular area of ​​the heavy equipment M1 and the center of the rectangular area of ​​the person P2 is greater than a threshold, the person P2 is excluded from the gaze target. Furthermore, in the example of FIG. 19 , before time Ta, the degree of overlap between the rectangular areas is less than a threshold, so the peripheral object is not selected as the gaze target.

[0088] As described above, the gaze target may be determined based on the positional relationship between a related object related to the behavior to be determined and a peripheral object. This allows, for example, when a dangerous situation occurs, such as unsafe behavior, for a person engaging in dangerous behavior to be set as the gaze target. Therefore, the image quality of the area including the person can be ensured, and behavior such as unsafe behavior can be recognized with high accuracy.

[0089] It should be noted that the present disclosure is not limited to the above-described embodiment and can be modified as appropriate without departing from the spirit of the present disclosure. For example, in the above-described embodiment, the center server extracts the gaze target and the terminal determines the gaze area based on the extracted gaze target. However, the center server may determine the gaze area based on the extracted gaze target. In this case, the center server may notify the terminal of the coordinates and size of the gaze area.

[0090] The processing flow described in the above embodiment is merely an example, and the order of the processing is not limited to the above example. The order of some of the processing may be changed, or some of the processing may be performed in parallel.

[0091] Each component in the above-described embodiments may be configured with hardware or software, or both, and may be configured with a single piece of hardware or software, or may be configured with multiple pieces of hardware or software. Each device and each function (processing) may be realized by a computer 30 having a processor 31 such as a CPU (Central Processing Unit) and a memory 32 serving as a storage device, as shown in FIG. 20 . For example, a program for performing the method (video processing method) in the embodiment may be stored in the memory 32, and each function may be realized by the processor 31 executing the program stored in the memory 32.

[0092] These programs include instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The programs may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0093] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0094] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A video processing system comprising: image quality control means for controlling the image quality of a gaze area including a gaze target in an input video; recognition means for performing a recognition process for an object included in the video for which the image quality of the gaze area has been controlled; and extraction means for extracting the gaze target based on a temporal change in a recognition result of the recognition process. (Supplementary Note 2) The video processing system according to Supplementary Note 1, wherein the recognition process includes a process for recognizing an action of the object, and the extraction means extracts the gaze target based on a temporal change in a score of an action label included in the recognition result. (Supplementary Note 3) The video processing system according to Supplementary Note 2, wherein the extraction means extracts the gaze target based on a temporal increase in the score of the action label. (Supplementary Note 4) The video processing system according to Supplementary Note 2, wherein the extraction means extracts the gaze target based on a temporal variation in the score of the action label. (Supplementary Note 5) The video processing system according to any one of Supplementary Notes 2 to 4, wherein the extraction means extracts the gaze target based on a positional relationship between a related object related to the behavior to be recognized and another object. (Supplementary Note 6) The video processing system according to Supplementary Note 5, wherein the pre-extraction means determines the other object to be the gaze target when a distance between the related object and the other object becomes closer than a predetermined threshold. (Supplementary Note 7) The video processing system according to Supplementary Note 5, wherein the extraction means determines the other object to be the gaze target when an overlap between a region including the related object and a region including the other object becomes greater than a predetermined threshold. (Supplementary Note 8) A video processing method comprising: controlling the image quality of a gaze region including the gaze target in an input video; performing recognition processing on objects included in the video in which the image quality of the gaze region has been controlled; and extracting the gaze target based on a temporal change in the recognition result of the recognition processing. (Supplementary Note 9) The video processing method according to Supplementary Note 8, wherein the recognition process includes a process of recognizing a behavior of the object, and the gaze target is extracted based on a temporal change in a score of an behavior label included in the recognition result. (Supplementary Note 10) The video processing method according to Supplementary Note 9, wherein the gaze target is extracted based on an amount of increase over time in the score of the behavior label.(Supplementary Note 11) The video processing method according to Supplementary Note 9, wherein the gaze target is extracted based on temporal variation in the score of the behavior label. (Supplementary Note 12) The video processing method according to any one of Supplementary Notes 9 to 11, wherein the gaze target is extracted based on a positional relationship between a related object related to the behavior to be recognized and another object. (Supplementary Note 13) The video processing method according to Supplementary Note 12, wherein when a distance between the related object and the other object becomes closer than a predetermined threshold, the other object is determined to be the gaze target. (Supplementary Note 14) The video processing method according to Supplementary Note 12, wherein when an overlap between a region including the related object and a region including the other object increases beyond a predetermined threshold, the other object is determined to be the gaze target. (Supplementary Note 15) A video processing device comprising: an image quality control means for controlling the image quality of a gaze area including a gaze target in an input video; a recognition means for performing a recognition process on an object included in the video for which the image quality of the gaze area has been controlled; and an extraction means for extracting the gaze target based on a temporal change in a recognition result of the recognition process. (Supplementary Note 16) The video processing device according to Supplementary Note 15, wherein the recognition process includes a process for recognizing an action of the object, and the extraction means extracts the gaze target based on a temporal change in a score of an action label included in the recognition result. (Supplementary Note 17) The video processing device according to Supplementary Note 16, wherein the extraction means extracts the gaze target based on a temporal increase in the score of the action label. (Supplementary Note 18) The video processing device according to Supplementary Note 16, wherein the extraction means extracts the gaze target based on a temporal variation in the score of the action label. (Supplementary Note 19) The video processing device according to any one of Supplementary Notes 16 to 18, wherein the extraction means extracts the gaze target based on a positional relationship between a related object related to the behavior to be recognized and another object. (Supplementary Note 20) The video processing device according to Supplementary Note 19, wherein the extraction means determines the other object to be the gaze target when a distance between the related object and the other object becomes closer than a predetermined threshold, or when an overlap between an area including the related object and an area including the other object increases beyond a predetermined threshold.

[0095] 1 Remote monitoring system 10 Video processing system 11 Image quality control unit 12 Recognition unit 13 Extraction unit 20 Video processing device 30 Computer 31 Processor 32 Memory 100 Terminal 101 Camera 102 Compression efficiency optimization function 103 Video transmission function 110 Video acquisition unit 120 Detection unit 130 Image quality change determination unit 131 First determination unit 132 Second determination unit 140 Compression efficiency determination unit 150 Terminal communication unit 200 Center server 201 Video recognition function 202 Alert generation function 203 GUI drawing function 204 Screen display function 210 Center communication unit 220 Decoder 230 Behavior recognition unit 231 Object detection unit 232 Tracking unit 233 Behavior predictor 234 Behavior determination unit 240 Analysis information storage unit 250 Gaze target analysis unit 300 Base station 400 MEC 401 Compression bit rate control function 402 Terminal control function

Claims

1. an image quality control means for controlling the image quality of a gaze area including a gaze target in an input image; A recognition means for performing a recognition process for an object included in the image in which the image quality of the gaze area is controlled; an extraction means for extracting the gaze target based on a temporal change in a recognition result of the recognition processing; An image processing system comprising:

2. The recognition process includes a process of recognizing a behavior of the object, The extraction means extracts the gaze target based on a temporal change in a score of an action label included in the recognition result. The video processing system according to claim 1 .

3. The extraction means extracts the gaze target based on a time increase in a score of the behavior label. The video processing system according to claim 2 .

4. The extraction means extracts the gaze target based on a time variation in the score of the behavior label. The video processing system according to claim 2 .

5. The extraction means extracts the gaze target based on a positional relationship between a related object related to the behavior to be recognized and another object. The image processing system according to claim 2 .

6. The extraction means determines the other object as the gaze target when a distance between the related object and the other object becomes shorter than a predetermined threshold. The video processing system according to claim 5 .

7. When an overlap between a region including the related object and a region including the other object becomes greater than a predetermined threshold, the extraction means determines the other object as the gaze target. The video processing system according to claim 5 .

8. Controlling the image quality of a gaze area including a gaze target in an input image; performing a recognition process for an object included in the image in which the image quality of the gaze area has been controlled; extracting the gaze target based on a temporal change in a recognition result of the recognition processing; Image processing method.

9. an image quality control means for controlling the image quality of a gaze area including a gaze target in an input image; A recognition means for performing a recognition process for an object included in the image in which the image quality of the gaze area is controlled; an extraction means for extracting the gaze target based on a temporal change in a recognition result of the recognition processing; The image processing device includes:

10. Controlling the image quality of a gaze area including a gaze target in an input image, performing a recognition process for an object included in the image in which the image quality of the gaze area has been controlled; extracting the gaze target based on a temporal change in a recognition result of the recognition processing; An image processing program that causes a computer to execute the processing.