Image processing system, image processing method, image processing device, and image processing program

Abstract

A video processing system (10) comprising: an image quality control unit (11) that controls the image quality in a gaze region including a gaze subject in a video that is input into the video processing system (10); a recognition unit (12) that performs a recognition process related to an object that is included in the video having the image quality in the gaze region controlled by the image quality control unit (11); and an extraction unit (13) that, on the basis of uncertainty of the recognition results from the recognition process performed by the recognition unit (12), extracts the gaze subject included in the gaze region that the image quality control unit (11) controls.

Description

Image processing system, image processing method, and image processing device

[0001] The present disclosure relates to a video processing system, a video processing method, and a video processing device.

[0002] Development is underway for systems that perform surveillance and other functions by applying machine learning-based detection and recognition technologies to video footage captured by cameras.

[0003] A related technique is known, for example, from Patent Document 1. Patent Document 1 describes a remote monitoring system in which video captured by a camera mounted on a vehicle is transmitted via a network and analyzed at a remote monitoring center, in which an area including a target object detected from the video is made high quality and other areas are made low quality before transmission.

[0004] International Publication No. 2022 / 074700

[0005] In related technologies such as that disclosed in Patent Document 1, the image quality of the area including the detected target object is always controlled to be high. For this reason, it may be difficult to appropriately reduce the amount of image data in the related technologies.

[0006] In view of the above problems, an object of the present disclosure is to provide a video processing system, a video processing method, and a video processing device that are capable of appropriately controlling the amount of video data.

[0007] The image processing system according to the present disclosure includes an image quality control means for controlling the image quality of a gaze area including a gaze target in an input image, a recognition means for performing a recognition process for an object included in the image in which the image quality of the gaze area has been controlled, and an extraction means for extracting the gaze target based on the uncertainty of the recognition result of the recognition process.

[0008] The image processing method according to the present disclosure controls the image quality of a gaze area in an input image that includes a gaze target, performs recognition processing on an object included in the image in which the image quality of the gaze area has been controlled, and extracts the gaze target based on the uncertainty of the recognition result of the recognition processing.

[0009] The video processing device according to the present disclosure includes an image quality control means for controlling the image quality of a gaze area including a gaze target in an input video, a recognition means for performing a recognition process for an object included in the video in which the image quality of the gaze area has been controlled, and an extraction means for extracting the gaze target based on the uncertainty of the recognition result of the recognition process.

[0010] According to the present disclosure, it is possible to provide a video processing system, a video processing method, and a video processing device that are capable of appropriately controlling the amount of video data.

[0011] 1 is a configuration diagram showing an overview of a video processing system according to an embodiment. FIG. 2 is a configuration diagram showing an overview of a video processing device according to an embodiment. FIG. 3 is a flowchart showing an overview of a video processing method according to an embodiment. FIG. 4 is a configuration diagram showing a basic configuration of a remote monitoring system. FIG. 5 is a configuration diagram showing an example of a configuration of a terminal according to embodiment 1. FIG. 6 is a configuration diagram showing an example of a configuration of a center server according to embodiment 1. FIG. 7 is a configuration diagram showing an example of a configuration of a behavior recognition unit according to embodiment 1. FIG. 8 is a configuration diagram showing an example of a configuration of a predictor according to embodiment 1. FIG. 9 is a flowchart showing an example of an operation of a remote monitoring system according to embodiment 1. FIG. 10 is a diagram for explaining video acquisition processing according to embodiment 1. FIG. 11 is a diagram for explaining object detection processing according to embodiment 1. FIG. 12 is a flowchart showing an example of an operation of behavior recognition processing according to embodiment 1. FIG. 13 is a flowchart showing an operation example of gaze target extraction processing according to embodiment 1. FIG. 14 is a diagram for explaining an example of recognition results with variation in the gaze target extraction processing according to embodiment 1. FIG. 15 is a diagram for explaining another example of recognition results with variation in the gaze target extraction processing according to embodiment 1. FIG. 16 is a diagram for explaining an example of recognition results without variation in the gaze target extraction processing according to embodiment 1. FIG. 17 is a diagram for explaining another example of recognition results without variation in the gaze target extraction processing according to embodiment 1. Fig. 1 is a diagram for explaining a gaze area determination process according to embodiment 1. Fig. 2 is a configuration diagram showing a configuration example of a behavior recognition unit according to embodiment 2. Fig. 3 is a flowchart showing an operation example of behavior recognition process according to embodiment 2. Fig. 4 is a diagram for explaining dropout process according to embodiment 2. Fig. 5 is a diagram for explaining dropout process according to embodiment 2. Fig. 6 is a configuration diagram showing an overview of hardware of a computer according to an embodiment.

[0012] Hereinafter, embodiments will be described with reference to the drawings. In the drawings, the same elements are denoted by the same reference numerals, and redundant description will be omitted as necessary.

[0013] In a system that collects video via a network and recognizes objects, actions, etc. in the video, it is preferable to minimize the amount of video data to be transmitted, since the bandwidth of the network that transmits the video is limited. For example, the amount of video data can be reduced by increasing the video compression rate. However, if the video compression rate is high or if the data loss rate is high, the number of false recognitions increases, resulting in a decrease in recognition accuracy. Therefore, in the embodiment, it is possible to prevent false recognition while minimizing the amount of video data to be transmitted. For example, false recognition of important events, such as unsafe or dangerous behavior, can be prevented.

[0014] (Outline of the embodiment) First, an outline of the embodiment will be described. Fig. 1 shows a schematic configuration of a video processing system 10 according to the embodiment. The video processing system 10 can be applied to, for example, a remote monitoring system that collects video via a network and monitors the video.

[0015] As shown in FIG. 1, the video processing system 10 includes an image quality control unit 11, a recognition unit 12, and an extraction unit 13.

[0016] The image quality control unit 11 controls the image quality of a gaze area including a gaze target in an input video. For example, the image quality control unit 11 may improve the image quality of the gaze area, i.e., make it clearer, than other areas. The recognition unit 12 performs recognition processing on an object included in the video in which the image quality of the gaze area has been controlled. For example, the object recognition processing is behavior recognition processing that recognizes the behavior of the object, but it may also be processing that recognizes information or features related to other objects.

[0017] The extraction unit 13 extracts the gaze target based on the uncertainty of the recognition result of the recognition process performed by the recognition unit 12. The uncertainty of the recognition result refers to the variability of the recognition result. For example, the recognition unit 12 may include multiple behavior predictors that have learned different learning data, and the extraction unit 13 may extract the gaze target based on the variability of the recognition results of multiple behaviors output from the multiple behavior predictors. In this case, the gaze target may be extracted based on the behavior labels included in the recognition results of the behaviors or the variability of the behavior label scores. Alternatively, the recognition unit 12 may include one behavior predictor, and the extraction unit 13 may extract the gaze target based on the variability of the recognition results of multiple behaviors output by the single behavior predictor when the single behavior predictor performs behavior recognition multiple times. Extracting the gaze target means selecting the gaze target from candidates included in the recognition result.

[0018] The video processing system 10 may be configured with one device or multiple devices. FIG. 2 shows the configuration of a video processing device 20 according to an embodiment. As shown in FIG. 2, the video processing device 20 may include the image quality control unit 11, the recognition unit 12, and the extraction unit 13 shown in FIG. 1. Furthermore, part or all of the video processing system 10 may be located at an edge or in the cloud. For example, in a system that monitors video captured at a site via a network, an edge is a device located at or near the site, and is also a device close to the terminal in terms of the network hierarchy. For example, the image quality control unit 11 may be located at an edge terminal, and the recognition unit 12 and extraction unit 13 may be located on a cloud server. Furthermore, each function may be distributed to the cloud.

[0019] Fig. 3 shows a video processing method according to an embodiment. For example, the video processing method according to the embodiment is executed by the video processing system 10 of Fig. 1 or the video processing device 20 of Fig. 2. As shown in Fig. 3, first, the image quality of a gaze area including a gaze target in an input video is controlled (S11). Next, a recognition process is performed on an object included in the video in which the image quality of the gaze area has been controlled (S12). Next, the gaze target is extracted based on the uncertainty of the recognition result of the recognition process (S13). Then, returning to S11, the image quality of the gaze area including the extracted gaze target is controlled for the input video.

[0020] As described above, the video processing system according to the embodiment recognizes the behavior of an object from video with controlled image quality, and extracts a gaze target based on the uncertainty of the recognition results. For example, if the recognition results vary, the behavior may not be correctly recognized, and the reliability of the recognition results may be low. Therefore, by making the object corresponding to the recognition result the gaze target and sharpening it, the behavior can be correctly recognized. Furthermore, since areas other than the target to be gazed at can be compressed, the amount of video data to be transmitted can be reduced.

[0021] (Basic Configuration of Remote Monitoring System) Next, a remote monitoring system, which is an example of a system to which the embodiment is applied, will be described. FIG. 4 shows the basic configuration of the remote monitoring system 1. The remote monitoring system 1 is a system that monitors an area captured by a camera using video. In this embodiment, the system will be described as a system that remotely monitors the work of workers at a site. For example, the site may be an area where people and machines are operating, such as a work site such as a construction site, a public square where people gather, or a school. In this embodiment, the work will be described as construction work, civil engineering work, etc., but is not limited to this. Note that video includes multiple images (also referred to as frames) in a time series, so the terms video and image are interchangeable. In other words, the remote monitoring system can be considered a video processing system that processes video and an image processing system that processes images.

[0022] As shown in FIG. 4 , the remote monitoring system 1 includes multiple terminals 100, a center server 200, a base station 300, and an MEC 400. The terminals 100, the base station 300, and the MEC 400 are located on the field side, and the center server 200 is located on the center side. For example, the center server 200 is located in a data center, a monitoring center, or the like located away from the field. The field side is the edge side of the system, and the center side is also the cloud side. The center server 200 may be configured as a single device or multiple devices. Furthermore, part or all of the center server 200 may be located in the cloud. For example, the image recognition function 201 and the alert generation function 202 may be located in the cloud, and the GUI drawing function 203 and the screen display function 204 may be located in a monitoring center, or the like.

[0023] The terminal 100 and the base station 300 are communicatively connected via a network NW1. The network NW1 is, for example, a wireless network such as 4G, local 5G / 5G, LTE (Long Term Evolution), or wireless LAN. The base station 300 and the center server 200 are communicatively connected via a network NW2. The network NW2 includes, for example, a core network such as 5GC (5th Generation Core network) or EPC (Evolved Packet Core), or the Internet. It can also be said that the terminal 100 and the center server 200 are communicatively connected via the base station 300. The base station 300 and the MEC 400 are communicatively connected via any communication method, but the base station 300 and the MEC 400 may be a single device.

[0024] The terminal 100 is a terminal device connected to the network NW1 and also serves as an image generating device that generates images of the site. The terminal 100 acquires images captured by a camera 101 installed at the site and transmits the acquired images to the center server 200 via the base station 300. The camera 101 may be located outside the terminal 100 or inside the terminal 100.

[0025] The terminal 100 compresses video from the camera 101 to a predetermined bit rate and transmits the compressed video. The terminal 100 has a compression efficiency optimization function 102 that optimizes compression efficiency, and a video transmission function 103. The compression efficiency optimization function 102 performs ROI control that controls the image quality of an ROI (Region of Interest; also called a fixation region). The compression efficiency optimization function 102 reduces the bit rate by maintaining the image quality of an ROI that includes a person or object while lowering the image quality of the surrounding area. The video transmission function 103 transmits the video with controlled image quality to the center server 200.

[0026] The base station 300 is a base station device of the network NW1, and also a relay device that relays communication between the terminal 100 and the center server 200. For example, the base station 300 is a local 5G base station, a 5G gNB (next generation Node B), an LTE eNB (evolved Node B), a wireless LAN access point, or the like, but may also be another relay device.

[0027] The MEC (Multi-access Edge Computing) 400 is an edge processing device located on the edge side of the system. The MEC 400 is an edge server that controls the terminal 100 and has a compression bit rate control function 401 and a terminal control function 402 that control the bit rate of the terminal. The compression bit rate control function 401 controls the bit rate of the terminal 100 through adaptive video distribution control and QoE (quality of experience) control. For example, the compression bit rate control function 401 predicts the recognition accuracy to be obtained while suppressing the bit rate according to the communication environment of the networks NW1 and NW2, and allocates a bit rate to the camera 101 of each terminal 100 so as to improve the recognition accuracy. The terminal control function 402 controls the terminal 100 to transmit video at the allocated bit rate. The terminal 100 encodes the video so as to obtain the allocated bit rate and transmits the encoded video.

[0028] The center server 200 is a server installed on the center side of the system. The center server 200 may be one or more physical servers, or may be a cloud server built on the cloud or other virtualized server. The center server 200 is a monitoring device that monitors on-site work by recognizing the work of people from on-site camera images. The center server 200 is also a recognition device that recognizes the behavior of people in the images transmitted from the terminal 100.

[0029] The center server 200 has an image recognition function 201, an alert generation function 202, a GUI drawing function 203, and a screen display function 204. The image recognition function 201 inputs the image transmitted from the terminal 100 into an image recognition AI (Artificial Intelligence) engine to recognize the work performed by the worker, i.e., the type of human behavior. The alert generation function 202 generates an alert according to the recognized work. The GUI drawing function 203 displays a GUI (Graphical User Interface) on the screen of the display device. The screen display function 204 displays the image of the terminal 100, the recognition results, alerts, etc. on the GUI.

[0030] (Embodiment 1) Next, a description will be given of embodiment 1. In this embodiment, an example will be described in which a gaze target is extracted based on the variation in behavior recognition results obtained by a plurality of predictors.

[0031] First, the configuration of the remote monitoring system according to this embodiment will be described. The basic configuration of the remote monitoring system 1 according to this embodiment is as shown in FIG. 4. Here, an example configuration of the terminal 100 and the center server 200 will be described. FIG. 5 shows an example configuration of the terminal 100 according to this embodiment, and FIG. 6 shows an example configuration of the center server 200 according to this embodiment. Note that the configuration of each device is merely an example, and other configurations may be used as long as the operations according to this embodiment, which will be described later, are possible. For example, some of the functions of the terminal 100 may be arranged in the center server 200 or another device, or some of the functions of the center server 200 may be arranged in the terminal 100 or another device.

[0032] As shown in FIG. 5, the terminal 100 includes a video acquisition unit 110 , a detection unit 120 , an image quality change determination unit 130 , a compression efficiency determination unit 140 , and a terminal communication unit 150 .

[0033] The video acquisition unit 110 acquires video (also referred to as input video) captured by the camera 101. For example, the input video may include a person who is a worker performing work on-site, a work object used by the person, etc. The video acquisition unit 110 also serves as an image acquisition unit that acquires multiple images in time series.

[0034] The detection unit 120 is an object detection unit that detects objects in the acquired input video. The detection unit 120 detects objects in each image included in the input video and assigns labels to the detected objects, i.e., object labels. An object label is an object class and indicates the type of object. For example, object labels include person, car, robot, hammer, etc. The detection unit 120 extracts a rectangular area containing an object from each image included in the input video, recognizes the object within the extracted rectangular area, and assigns a label to the recognized object. The rectangular area is a bounding box or an object area. Note that the object area containing the object is not limited to a rectangular area and may be a circular or irregular silhouette area, etc. The detection unit 120 calculates image features of the object contained in the rectangular area and recognizes the object based on the calculated features. For example, the detection unit 120 recognizes objects in the image using an object recognition engine that uses machine learning such as deep learning. Objects can be recognized by machine learning the image features and object labels of the object. The object detection result includes the object label, position information of the rectangular area containing the object, etc. The object position information is, for example, the coordinates of each vertex of a rectangular area, but may also be the position of the center of the rectangular area or the position of any point on the object. The detection unit 120 transmits the object detection result to the image quality change determination unit 130.

[0035] The image quality change determination unit 130 determines a region of interest (ROI), which is an image quality change region in which the image quality of the acquired input video is changed. The image quality change determination unit 130 is a determination unit that determines the region of interest. The region of interest is a region that includes an object of attention, and is a region in which the image quality is improved, i.e., sharpened. The region of interest can also be said to be a region that ensures image quality for behavior recognition.

[0036] For example, the image quality change determination unit 130 includes a first determination unit 131 and a second determination unit 132. For example, the first determination unit 131 first determines a gaze area, and after the center server 200 recognizes the behavior, the second determination unit 132 determines the gaze area. Note that the determination of the gaze area by the first determination unit 131 may be omitted, and only the determination of the gaze area by the second determination unit 132 may be performed. The first determination unit 131 determines the gaze area of ​​the input video based on the detection result of an object detected in the input video by the detection unit 120. The first determination unit 131 determines the gaze area based on position information of an object that has a label indicating the gaze target among the detected objects detected in the input video by the detection unit 120. The gaze target may be a person who is the target of behavior recognition, or may include a work object that the person can use in their work. For example, the label of the work object is preset as the label of an object associated with the person. Note that the target of behavior recognition is not limited to a person, but may also be an object such as heavy machinery or a robot. That is, actions including work performed by heavy machinery or robots may be recognized.

[0037] When information is fed back from the center server 200 that recognized the behavior, the second determination unit 132 determines the gaze area of ​​the input video based on the fed back information. In this example, extracted gaze target information, which is information on the gaze target extracted by the center server 200, is fed back. The extracted gaze target information is information on the gaze target, and is information indicating the gaze target extracted by the center server 200 through behavior recognition. The extracted gaze target information is position information of the gaze target, and includes position information of a rectangular area of ​​the gaze target. For example, the second determination unit 132 determines the rectangular area indicated by the acquired extracted gaze target information as the gaze area. In other words, the second determination unit 132 determines an area that ensures the image quality of the input video based on the position of the extracted gaze target.

[0038] The compression efficiency determination unit 140 determines a compression rate for the attention area or an area other than the attention area and compresses the video. The compression efficiency determination unit 140 is an encoder that encodes the input video using the determined compression rate. The compression efficiency determination unit 140 encodes the input video using a video encoding method such as H.264 or H.265. The compression efficiency determination unit 140 also encodes the input video so that the bit rate is equal to the bit rate assigned by the compression bit rate control function 401 of the MEC 400.

[0039] The compression efficiency determination unit 140 is an image quality control unit that controls the image quality of the attention area determined by the image quality change determination unit 130 and corresponds to the image quality control unit 11 in FIG. 1 . The compression efficiency determination unit 140 can also be considered an image quality improvement unit that improves the image quality of the attention area. The attention area is an area determined by either the first determination unit 131 or the second determination unit 132. The compression efficiency determination unit 140 compresses the attention area and other areas at predetermined compression rates, respectively, to encode the image quality of the attention area to a predetermined quality. In other words, by changing the compression rates of the attention area and other areas, the image quality of the attention area is improved compared to other areas. It can also be said that the image quality of other areas is reduced compared to the attention area. For example, the image quality of the attention area and other areas is controlled within the bit rate range assigned by the compression bit rate control function 401 of the MEC 400. The image quality of the attention area may be controlled not only by the compression rate but also by changing the image resolution, frame rate, etc. The image quality of the gaze area may also be controlled by changing the amount of color information in the image, for example, color, grayscale, black and white, etc.

[0040] The terminal communication unit 150 transmits the encoded data encoded by the compression efficiency determination unit 140 to the center server 200 via the base station 300. The terminal communication unit 150 is a transmission unit that transmits video in which the image quality of the gaze area has been controlled. The terminal communication unit 150 also receives extracted gaze target information transmitted from the center server 200 via the base station 300. The terminal communication unit 150 is an acquisition unit that acquires the extracted gaze target information. The terminal communication unit 150 is an interface capable of communicating with the base station 300, and is, for example, a wireless interface such as 4G, local 5G / 5G, LTE, or wireless LAN, but may also be a wireless or wired interface of any other communication method. The terminal communication unit 150 may include a first terminal communication unit that transmits the encoded data and a second terminal communication unit that receives the extracted gaze target information. The first terminal communication unit and the second terminal communication unit may be communication units of the same communication method or communication units of different communication methods.

[0041] As shown in FIG. 6 , the center server 200 includes a center communication unit 210 , a decoder 220 , a behavior recognition unit 230 , an analysis information storage unit 240 , and a gaze target analysis unit 250 .

[0042] The center communication unit 210 receives encoded data transmitted from the terminal 100 via the base station 300. The center communication unit 210 is a receiving unit that receives video in which the image quality of the gaze area has been controlled. The center communication unit 210 also transmits extracted gaze target information extracted by the gaze target analysis unit 250 to the terminal 100 via the base station 300. The center communication unit 210 is a notifying unit that notifies the extracted gaze target information. The center communication unit 210 is an interface capable of communicating with the Internet or a core network, and is, for example, a wired interface for IP communication, but may also be a wired or wireless interface of any other communication method. The center communication unit 210 may include a first center communication unit that receives encoded data and a second center communication unit that transmits extracted gaze target information. The first center communication unit and the second center communication unit may be communication units using the same communication method or different communication methods.

[0043] The decoder 220 decodes the encoded data received from the terminal 100. The decoder 220 decodes using a video encoding method compatible with the encoding method of the terminal 100, such as H.264 or H.265. The decoder 220 performs decoding according to the compression rate of each region and generates decoded video (also referred to as received video).

[0044] The behavior recognition unit 230 is a recognition unit that recognizes the behavior of an object in the decoded received video, and corresponds to the recognition unit 12 in FIG. 1. The behavior recognition unit 230 executes behavior recognition processing to recognize the behavior of the gaze target for the video in which the image quality of the gaze area has been controlled. The behavior recognition unit 230 detects an object from the received video and recognizes the behavior of the detected object. The behavior recognition unit 230 recognizes the behavior of a person who is the target of behavior recognition and assigns a label to the recognized behavior, i.e., a behavior label. The behavior label is a class of behavior and indicates the type of behavior.

[0045] For example, the behavior recognition unit 230 recognizes a person's behavior based on the person and work object detected from the received video. The behavior recognition unit 230 may recognize a person's behavior by identifying the relationship between the person and the work object. The relationship between the person and the work object includes which object the person is using or not using. For example, a work object for each person may be identified based on the distance between the person and the work object, and behavior may be recognized from the identified work object. For example, the behavior recognition unit 230 performs machine learning on the work object and task associated with the person, and recognizes the person's behavior based on machine learning. Note that, without being limited to machine learning, the behavior recognition unit 230 may associate the work object and task associated with the person and recognize the person's behavior based on rules. For example, the behavior recognition unit 230 may associate the work object and task associated with the person in advance, and recognize the person's behavior based on the detected work object. Alternatively, if a work object is not detected, behavior may be recognized from the person alone. For example, the behavior recognition unit 230 may associate the person's posture and shape with the task in advance, and recognize the person's behavior based on the detected posture and shape. In this embodiment, the behavior recognition unit 230 includes a plurality of predictors that respectively predict behaviors from received video, and outputs behavior recognition results predicted by the plurality of predictors.

[0046] The analysis information storage unit 240 stores the analysis information analyzed by the behavior recognition unit 230. The analysis information includes behavior recognition results, person detection information, and work object detection information related to the behavior. Note that the behavior recognition results may include person detection information and work object detection information related to the behavior. The behavior recognition results include a label of the recognized behavior, a score of the behavior label, identification information of the person performing the recognized behavior, and identification information of the work object used in the recognized behavior. The behavior label score indicates the confidence level, which is the likelihood (probability) of the behavior label. The higher the score, the more likely the predicted behavior of the behavior label is to be correct. The person detection information includes position information of the person's rectangular area, tracking information, and the like. The tracking information is trajectory information indicating the tracking results of the object. The work object detection information includes an object label, an object label score, position information of the object's rectangular area, tracking information, and the like. For example, the behavior predictor (behavior recognition engine) of the behavior recognition unit 230 learns to weight objects related to the behavior, thereby extracting candidate work objects that may be related to each image and outputting information about the extracted candidate work objects. For example, if a pile driving operation is recognized, information about a hammer, which is an object related to the action, is output.

[0047] The gaze target analysis unit 250 is an extraction unit that extracts a gaze target based on the analysis information analyzed by the behavior recognition unit 230, and corresponds to the extraction unit 13 in FIG. 1 . The analysis information may be acquired from the behavior recognition unit 230 or from the analysis information storage unit 240. The gaze target analysis unit 250 determines a gaze target that ensures image quality to prevent behavior recognition errors based on the analyzed information. For example, the gaze target analysis unit 250 determines a gaze target based on the behavior recognition result. The gaze target analysis unit 250 determines a person whose behavior is recognized by the behavior recognition unit 230, i.e., a person whose behavior is included in the behavior recognition result, as the gaze target. When an action is recognized from a work object associated with a person, the person and the work object may be determined as the gaze target. There may be multiple work objects associated with a person, and the person and multiple work objects may be determined as the gaze targets. For example, when a pile driving operation is recognized, the objects related to the operation may be a "pile" and a "hammer," and the person, the "pile," and the "hammer" may be determined as the gaze targets.

[0048] In this embodiment, the gaze target analysis unit 250 extracts the gaze target based on the uncertainty of the behavior recognition result. For example, the gaze target is determined based on the variability of the multiple behavior recognition results output by the multiple predictors of the behavior recognition unit 230. The gaze target is determined based on the variability of the behavior labels included in the multiple behavior recognition results and the variability of the behavior label scores. For example, if the variability of the behavior recognition results is greater than a predetermined range, the gaze target may be determined to be an object including a person who performed the behavior of the behavior label and a working object.

[0049] The gaze target analysis unit 250 outputs position information of the extracted gaze target rectangular area as extracted gaze target information. The position information is, for example, the coordinates of each vertex of the rectangular area, but may also be the center position of the rectangular area or the position of any point of the gaze target. The extracted gaze target information is not limited to position information as information about the extracted gaze target, and may also include information analyzed by the behavior recognition unit 230, such as the object label of the gaze target, image features, behavior label, and behavior label score.

[0050] FIG. 7 shows an example configuration of the behavior recognition unit 230 in the center server 200. As shown in FIG. 7, the behavior recognition unit 230 includes multiple predictors PM1 to PM3. Note that the number of predictors PM is not limited to three, and any number of predictors PM may be included. The predictors PM1 to PM3 each predict the behavior of an object in received video, i.e., recognize the behavior. The predictors PM1 to PM3 may be learning models with the same configuration but trained using different training data sets. For example, the predictors PM1 to PM3 each train using video footage of the same behavior or the same type of behavior captured in different work sites (environments). The predictor learning models are object recognition engines or behavior recognition engines. Note that the predictors PM1 to PM3 may each be learning models that have trained different behaviors or different types of behavior. For example, one predictor may train a first behavior, such as behavior during excavation work, and another predictor may train a second behavior, such as behavior during ground leveling work. In this case, the training data for excavation work and the training data for ground leveling work may be videos taken in the same environment. The different actions that the predictors learn may be actions that can be performed simultaneously or actions that cannot be performed simultaneously.

[0051] Fig. 8 shows an example of the configuration of the predictor PM in Fig. 7. Fig. 8 shows an example of the configuration when behavior recognition based on the association between a person and a work object is performed based on machine learning. In the example of Fig. 8, the predictor PM of the behavior recognition unit 230 includes an object detection unit 231, a tracking unit 232, a behavior predictor 233, and a behavior determination unit 234.

[0052] The object detection unit 231 detects objects in the input received video. For example, like the detection unit 120 of the terminal 100, the object detection unit 231 is a detection unit such as an object recognition engine using machine learning. That is, the object detection unit 231 extracts a rectangular area including an object from each image of the received video, recognizes the object in the extracted rectangular area, and assigns a label to the recognized object. The object detection result includes the object label and position information of the rectangular area including the object.

[0053] The tracking unit 232 tracks the detected object in the received video. Based on the object detection result, the tracking unit 232 associates the object in each image included in the received video. By assigning a tracking ID to the detected object, each object can be identified and tracked. For example, the object is tracked by associating the object between images based on the distance or overlap (e.g., IoU: Intersection over Union) between the rectangular area of ​​the object detected in the previous image and the rectangular area of ​​the object detected in the next image.

[0054] The behavior predictor 233 predicts the behavior of each object tracked by the tracking unit 232. The behavior predictor 233 recognizes the behavior of the person tracked in the received video and assigns a label to the recognized behavior. For example, the behavior predictor 233 recognizes the behavior of the person in the received video using a behavior recognition engine that uses machine learning such as deep learning. The behavior of the person can be recognized by machine learning the video of the person performing work using the work object and the behavior label. For example, machine learning is performed using learning data that is a video of the person performing work using the work object, annotation information such as the positions of the person and the work object and information related to the person and the object, and behavior information such as the work object required for each work. The behavior predictor 233 also outputs a score for the recognized behavior label.

[0055] The behavior determination unit 234 determines the behavior of the object based on the predicted behavior label. The behavior determination unit 234 determines the behavior of the person based on the score of the behavior label predicted by the behavior predictor 233. For example, the behavior determination unit 234 outputs the behavior label with the highest score as the recognition result. Note that the recognition result may include the scores of multiple behavior labels predicted by the behavior predictor 233.

[0056] Next, the operation of the remote monitoring system according to this embodiment will be described. Fig. 9 shows an example of the operation of the remote monitoring system 1. For example, the explanation will be given assuming that the terminal 100 executes S101 to S105 and S111 to S112, and the center server 200 executes S106 to S110, but this is not limiting and any device may execute each process.

[0057] 9, the terminal 100 acquires video from the camera 101 (S101). The camera 101 generates video of a work site, and the video acquisition unit 110 acquires the video (input video) output from the camera 101. For example, as shown in FIG. 10, the image of the input video includes a person working on the work site and a work object such as a hammer used by the person.

[0058] Next, the terminal 100 detects objects based on the acquired input video (S102). The detection unit 120 uses an object recognition engine to detect rectangular areas within an image included in the input video, recognize objects within the detected rectangular areas, and assign labels to the recognized objects. For each detected object, the detection unit 120 outputs the object label and position information of the object's rectangular area as the object detection result. For example, when object detection is performed from the image of FIG. 10, a person and a hammer are detected, and the rectangular area of ​​the person and the rectangular area of ​​the hammer are detected, as shown in FIG. 11.

[0059] Next, the terminal 100 determines a gaze area in the input video based on the object detection result (S103). The first determination unit 131 of the image quality change determination unit 130 extracts objects having a label that is the gaze target based on the object detection result for each object. The first determination unit 131 extracts objects whose object label is person or work object from the detected objects, and determines the rectangular area of ​​the corresponding object as the gaze area. In the example of Figure 11, a person and a hammer are detected in the image, and the hammer corresponds to the work object, so the rectangular area of ​​the person and the rectangular area of ​​the hammer are determined as the gaze area.

[0060] Next, the terminal 100 encodes the input video based on the determined gaze area (S104). The compression efficiency determination unit 140 encodes the input video so that the gaze area has higher image quality than other areas. In the example of Fig. 11, the compression rates of the rectangular area of ​​the person and the rectangular area of ​​the hammer are set lower than the compression rates of other areas, thereby improving the image quality of the rectangular area of ​​the person and the rectangular area of ​​the hammer.

[0061] Next, the terminal 100 transmits the encoded data to the center server 200 (S105), and the center server 200 receives the encoded data (S106). The terminal communication unit 150 transmits the encoded data with the image quality of the gaze area enhanced to the base station 300. The base station 300 transfers the received encoded data to the center server 200 via the core network or the Internet. The center communication unit 210 receives the transferred encoded data from the base station 300.

[0062] Next, the center server 200 decodes the received encoded data (S107). The decoder 220 decodes the encoded data in accordance with the compression rate of each region, and generates a video (received video) in which the image quality of the attention region is improved.

[0063] Next, the center server 200 recognizes the behavior of the object based on the decoded received video (S108). The predictors PM1 to PM3 of the behavior recognition unit 230 each analyze the received video and recognize the behavior of the object. Fig. 12 shows an example of behavior recognition processing by the predictor PM of the behavior recognition unit 230 shown in Fig. 8.

[0064] 12 , first, the object detection unit 231 detects objects in the input received video (S201). Using an object recognition engine, the object detection unit 231 detects rectangular areas in each image included in the received video, recognizes objects in the detected rectangular areas, and assigns labels to the recognized objects. For each detected object, the object detection unit 231 outputs the object label and position information of the object's rectangular area as object detection results.

[0065] Next, the tracking unit 232 tracks the detected objects in the received video (S202). The tracking unit 232 assigns a tracking ID to each detected object and tracks the object identified by the tracking ID in each image.

[0066] Next, the behavior predictor 233 predicts the behavior of each tracked object (S203). The behavior predictor 233 uses a behavior recognition engine to predict the behavior of the person from the video including the tracked person and the work object. The behavior predictor 233 outputs the label of the predicted behavior and the score of each behavior label.

[0067] Next, the behavior determination unit 234 determines the behavior of the object based on the scores of the predicted behavior labels (S204). In the example of FIG. 13, a person and a hammer are detected by tracking. The behavior predictor 233 recognizes the person's behavior based on the video of the detected person and hammer, and outputs a score for each behavior label. For example, the score for pile driving is 0.8, the score for heavy machinery operation is 0.1, the score for unsafe behavior is 0.0, and the score for non-work is 0.1. Then, the behavior determination unit 234 determines that the person's behavior is pile driving because the score for pile driving is the highest. The behavior determination unit 234 outputs the determined behavior and the behavior score.

[0068] Returning to FIG. 9 , following the behavior recognition process, the center server 200 extracts a gaze target based on the analysis information analyzed by the behavior recognition process (S109). The gaze target analysis unit 250 sets the person whose behavior has been recognized as the gaze target, and if the recognized target includes a work object, the work object may also be included as the gaze target. For example, in the example of FIG. 13 , since the work of driving a pile is recognized from the person and the hammer, the person whose work has been recognized and the hammer may be set as the gaze target. The gaze target analysis unit 250 outputs gaze target extraction information including position information of the extracted gaze target.

[0069] In this embodiment, a gaze target is extracted based on the uncertainty of the behavior recognition results. FIG. 14 shows an example of the operation of the gaze target extraction process according to this embodiment. When the behaviors of multiple objects are recognized, the gaze target extraction process is performed on the recognition results of each object. In the example of FIG. 14 , first, the gaze target analysis unit 250 acquires prediction results from multiple predictors, i.e., behavior recognition results (S301). For example, the gaze target analysis unit 250 acquires behavior recognition results including behavior labels and scores predicted by the predictors PM1 to PM3 of the behavior recognition unit 230. For example, the gaze target analysis unit 250 acquires the behavior label with the highest score output by the behavior determination unit 234 of each predictor. Note that multiple behavior labels and scores output from the behavior predictor 233 of each predictor may be acquired, or any number of behavior labels may be acquired from the multiple behavior labels output from the behavior predictor 233 of each predictor. For example, the top three behavior labels with the highest scores may be acquired. When multiple behavior labels are acquired from each predictor, the gaze target is extracted based on the variance of the multiple behavior labels of each predictor.

[0070] Next, the gaze target analysis unit 250 determines the variation of the multiple behavior recognition results (S302). For example, the gaze target analysis unit 250 determines the variation of the behavior labels included in the behavior recognition results of the multiple predictors. The gaze target analysis unit 250 may determine whether or not there is variation in the behavior labels, or may determine the magnitude of the variation. Furthermore, the variation may be determined not only by the variation of the behavior labels, but also by including the scores of the behavior labels.

[0071] If it is determined that there is variation among the multiple behavior recognition results, the gaze target analysis unit 250 determines the target object performing the predicted behavior as the gaze target (S303). For example, if there is variation in the behavior labels, the gaze target analysis unit 250 determines the target object with that behavior label as the gaze target. FIG. 15 shows an example in which it is determined that there is variation in the recognition results when the predictors PM1 to PM3 have learned the same behavior. For example, the predictors PM1 to PM3 have learned the same work captured in different environments. In the example of FIG. 15, the behavior label of the prediction result of predictor PM1 is heavy machinery operation, the behavior label of the prediction result of predictor PM2 is cart transportation, and the behavior label of the prediction result of predictor PM3 is compaction work. Since the prediction results are different, it is determined that there is variation, and the predicted target object is determined as the gaze target.

[0072] Furthermore, if there is a large variation in the activity labels, the gaze target analysis unit 250 may determine the target object of the activity label as the gaze target. For example, if the number M of matching activity labels with the highest scores among N predictors is equal to or less than a threshold T, the predicted target object is determined as the gaze target. For example, if the activity label of the prediction result of predictor PM1 is cart transport, the activity label of the prediction result of predictor PM2 is compaction work, the activity label of the prediction result of predictor PM3 is cart transport, and the activity label of the prediction result of predictor PM4 is heavy machinery work, then N = 4, M = 2, and T = 3. Since M < T, it is determined that there is a large variation, and the predicted target object is determined as the gaze target.

[0073] FIG. 16 illustrates an example in which it is determined that there is variation in the recognition results when predictors PM1 to PM3 have learned different actions. In this example, predictors PM1 to PM3 have learned different actions that were captured in the same environment and that can be performed simultaneously. For example, predictor PM1 is a predictor that has learned the posture (skeleton) of a person's actions, predictor PM2 is a predictor that has learned the content (task) of the person's actions, and predictor PM3 is a predictor that has learned detailed information about inspection work, among the actions recognized by predictor PM2. As shown in FIG. 16 , in the posture recognition results of predictor PM1, the scores for standing, walking, and crouching are almost the same, so the posture of the target object is unknown. In the action recognition results of predictor PM2, the scores for inspection, cleaning, walking, and other tasks are almost the same, so the action of the target object is unknown. In this case, since the recognition results of predictors PM1 and PM2 each contain multiple behavioral labels with scores of the same magnitude or above a predetermined value, it is determined that the recognition results of predictors PM1 to PM3 vary, and the target object is determined to be the object of attention.

[0074] Furthermore, when multiple predictors are learning different types of behaviors that cannot be performed simultaneously, it is expected that only the predictor that has learned the target behavior will respond to the behavior of the target object. For example, in an example using predictors A and B, compaction and heavy machinery work cannot be performed simultaneously, and predictor A has learned compaction work, while predictor B has learned heavy machinery work. In this example, when a worker is performing compaction work, only predictor A is expected to respond, resulting in a high score for compaction work and a low score for heavy machinery work by predictor B. Therefore, if the scores of both predictors are high, it can be determined that the behaviors cannot be distinguished and there is a high level of uncertainty in recognition. Therefore, a high level of uncertainty can be determined based on whether there are multiple behavior classes above a certain score threshold between different predictors. For example, in the recognition results of predictor A, the score for compaction work is 0.8, and in the recognition results of predictor B, the score for heavy machinery work is 0.9. If the score threshold is set to 0.6, there will be two action classes that exceed the score threshold, and therefore it is determined that there is a high degree of uncertainty.

[0075] Furthermore, if it is determined that there is no variance among the multiple prediction results, the gaze target analysis unit 250 excludes the target object performing the predicted behavior from the gaze target (S304). That is, in this case, the gaze target analysis unit 250 does not select the object as the gaze target. For example, if there is no variance in the activity label, the gaze target analysis unit 250 does not select the target object with that activity label as the gaze target. Similar to FIG. 15, FIG. 17 illustrates an example in which it is determined that there is no variance in the recognition results when the predictors PM1 to PM3 have learned the same activity. Similarly to FIG. 15, for example, the predictors PM1 to PM3 have learned the same work captured in different environments. In the example of FIG. 17, the activity labels of the prediction results of the predictors PM1 to PM3 are all heavy machinery operation, and the prediction results are consistent, so it is determined that there is no variance, and the predicted target object is excluded from the gaze target.

[0076] Furthermore, if the variation in the activity label is small, the gaze target analysis unit 250 may exclude the target object with that activity label from the gaze targets. For example, if the number M of matching activity labels with the highest scores among N predictors is greater than a threshold T, the predicted target is excluded from the gaze targets. For example, if the activity label of the prediction result of predictor PM1 is "cart transport," the activity label of the prediction result of predictor PM2 is "cart transport," the activity label of the prediction result of predictor PM3 is "cart transport," and the activity label of the prediction result of predictor PM4 is "heavy machinery operation," then N = 4, M = 3, and T = 2. Since M > T, it is determined that the variation is small, and the predicted target object is excluded from the gaze targets.

[0077] 18 shows an example in which it is determined that there is no variation in the recognition results when the predictors PM1 to PM3 have learned different actions, similar to Fig. 16. As in Fig. 16, the predictors PM1 to PM3 have learned different actions that have been photographed in the same environment and that can be performed simultaneously. For example, the predictor PM1 is a predictor that has learned the posture (skeleton) of a person's action, the predictor PM2 is a predictor that has learned the content (task) of the person's action, and the predictor PM3 is a predictor that has learned detailed information about inspection work, among the actions recognized by the predictor PM2. As shown in FIG. 18 , in the posture recognition results of predictor PM1, only the standing score is high and the other scores are low, so the posture of the target object is predicted to be standing. In the action recognition results of predictor PM2, only the inspection score is high and the other scores are low, so the target object's action is predicted to be inspection. In the inspection task recognition results of predictor PM3, only the tool check score is high and the other scores are low, so the target object's inspection task is predicted to be tool check. This makes it possible to recognize that the person representing the target object is performing an inspection task while standing, checking for the presence or absence of tools. In this case, since each of the recognition results of predictors PM1 to PM3 has one action label with a score equal to or greater than a predetermined value, it is determined that the recognition results of predictors PM1 to PM3 are consistent, and the target object is excluded from the gaze target.

[0078] In addition, in an example where multiple predictors are learning different types of behavior that cannot be performed simultaneously, if there are no multiple behavior classes above a certain score threshold between different predictors, for example, if there is only one behavior class above the score threshold, it may be determined that there is little uncertainty.

[0079] 9 , following the gaze target extraction process, the center server 200 notifies the terminal 100 of the extracted gaze target information extracted by the gaze target extraction process (S110), and the terminal 100 acquires the extracted gaze target information (S111). The center communication unit 210 transmits the extracted gaze target information indicating the position of the extracted gaze target to the base station 300 via the Internet or core network. The base station 300 transfers the received extracted gaze target information to the terminal 100. The terminal communication unit 150 receives the transferred extracted gaze target information from the base station 300.

[0080] Next, the terminal 100 determines a gaze area based on the received extracted gaze target information (S112). The second determination unit 132 of the image quality change determination unit 130 determines the area indicated by the extracted gaze target information notified by the center server 200 as the gaze area. In the example of FIG. 19, the extracted gaze target information indicates a rectangular area of ​​the person and a rectangular area of ​​the hammer, and these areas are determined as the gaze area. Alternatively, a circumscribing area including the rectangular area of ​​the person and the rectangular area of ​​the hammer may be determined as the gaze area. This circumscribing area may be notified to the terminal 100 from the center server 200. Thereafter, S104 to S112 are repeated.

[0081] As described above, in this embodiment, in a system that recognizes the behavior of an object from video, a gaze target is extracted based on the uncertainty of the behavior recognition result, and the image quality of the area including the extracted gaze target is sharpened. For example, if the behavior recognition results vary, it is assumed that the object has not been correctly recognized from the video. Therefore, by making a judgment based on the uncertainty of the behavior recognition result, it is possible to appropriately select the target to gaze on. Therefore, depending on the behavior recognition result, the image quality of a specific part including the target to be gazed on can be guaranteed, and other areas can be compressed, thereby preventing behavior recognition errors while reducing the amount of data to be transmitted.

[0082] (Second Embodiment) Next, a second embodiment will be described. In this embodiment, an example in which uncertainty of a behavior recognition result is output from a behavior recognition unit will be described. The configuration other than the behavior recognition unit is the same as that of the first embodiment shown in FIGS. 5 and 6, and therefore description thereof will be omitted. Note that this embodiment can be implemented in combination with the first embodiment, and each configuration shown in the first embodiment may be used as appropriate.

[0083] Fig. 20 shows an example configuration of the behavior recognition unit 230 according to this embodiment. As shown in Fig. 20, the behavior recognition unit 230 according to this embodiment includes an object detection unit 231, a tracking unit 232, a behavior predictor 233, and a behavior determination unit 234, which are similar to the predictor PM in Fig. 8, and further includes a dropout setting unit 235 and a variance calculation unit 236. The dropout setting unit 235 sets dropout in the neural network of the behavior predictor 233. The variance calculation unit 236 calculates the prediction results obtained by multiple predictions by the behavior predictor 233 in which dropout is set, i.e., the variance of the recognition results obtained by multiple recognitions.

[0084] Fig. 21 shows an example of behavior recognition processing by the behavior recognition unit 230 shown in Fig. 20. Note that other operations are similar to those in Fig. 9 of embodiment 1. In the example of Fig. 21, first, as in embodiment 1, the object detection unit 231 detects an object in the input received video (S201), and the tracking unit 232 tracks the detected object in the received video (S202).

[0085] Next, the dropout setting unit 235 sets a dropout in the behavior predictor 233 (S211). For example, as shown in FIG. 22, the neural network of the behavior predictor 233 is assumed to include an input layer, a hidden layer (intermediate layer), and an output layer. In this example, the input layer includes multiple nodes, and the hidden layer includes multiple nodes. In this case, the dropout setting unit 235, for example, selects nodes in the hidden layer and inactivates the selected nodes, as shown in FIG. 23. Nodes selected randomly may be inactivated, or nodes may be selected and inactivated so as to achieve a predetermined dropout rate. Note that nodes in the input layer may be inactivated, not just the hidden layer.

[0086] With dropout set and predetermined nodes deactivated, the behavior predictor 233 predicts the behavior of the object as in the first embodiment (S203), and the behavior determination unit 234 determines the behavior of the object based on the score of the predicted behavior label (S204). The behavior determination unit 234 outputs the determined behavior recognition result. Furthermore, S211, S203, and S204 are repeated multiple times to perform behavior recognition using dropout multiple times. The multiple recognitions may be performed by sequentially repeating the processes of S211, S203, and S204, or by duplicating the series of processes of S211, S203, and S204 and performing them in parallel. However, in the multiple recognitions, the nodes to be deactivated in S211 are set differently each time.

[0087] Thereafter, the variation calculation unit 236 calculates the variation of the multiple behavior recognition results obtained by predicting the behavior multiple times (S212). As in the first embodiment, the variation calculation unit 236 may calculate the variation of the behavior labels included in the behavior recognition results, or may calculate the variation of the scores of the behavior labels. The variation calculation unit 236 may determine whether or not there is variation in the behavior labels and output the determination result. If the behavior labels of the multiple behavior recognition results are different, the variation calculation unit 236 outputs "there is variation" as the variation calculation result, and if the behavior labels of the multiple behavior recognition results are the same, the variation calculation unit 236 outputs "there is no variation" as the variation calculation result.

[0088] Furthermore, the variation calculation unit 236 may determine whether the variation of the activity labels of multiple behavior recognition results is greater than a threshold value and output the determination result. As in the first embodiment, for example, when the number M of matching activity labels with the highest scores among N behavior recognitions (inferences) is equal to or less than a threshold value T, "Variation Present" is output as the variation calculation result, and when the number M of matching activity labels with the highest scores is greater than the threshold value T, "No Variation" is output as the variation calculation result. The degree of variation, which is the ratio of the number M of matching activity labels with the highest scores to the N behavior recognitions, may be output as the variation calculation result. Note that, as in the first embodiment, the gazing target analysis unit 250 may calculate and determine the variation.

[0089] Thereafter, similarly to the first embodiment, the gaze target analysis unit 250 extracts a gaze target based on the variation calculation result calculated by the variation calculation unit 236 of the behavior recognition unit 230. For example, when the presence or absence of variation is output as the variation calculation result, if there is variation, the target object of the behavior recognition result is determined as the gaze target. When the degree of variation is output as the variation calculation result, the gaze target may be determined according to the result of comparing the degree of variation with a threshold.

[0090] As described above, a gaze target may be extracted using a single behavior predictor that outputs uncertainty in behavior prediction. Even in this case, as in the first embodiment, an appropriate gaze target can be selected, thereby reducing the amount of data to be transmitted and preventing behavior recognition errors. Furthermore, uncertainty in the recognition result can be determined without preparing multiple predictors.

[0091] It should be noted that the present disclosure is not limited to the above-described embodiment and can be modified as appropriate without departing from the spirit of the present disclosure. For example, in the above-described embodiment, the center server extracts the gaze target and the terminal determines the gaze area based on the extracted gaze target. However, the center server may determine the gaze area based on the extracted gaze target. In this case, the center server may notify the terminal of the coordinates and size of the gaze area.

[0092] The processing flow described in the above embodiment is merely an example, and the order of the processing is not limited to the above example. The order of some of the processing may be changed, or some of the processing may be performed in parallel.

[0093] Each component in the above-described embodiments may be configured with hardware or software, or both, and may be configured with a single piece of hardware or software, or may be configured with multiple pieces of hardware or software. Each device and each function (processing) may be realized by a computer 30 having a processor 31 such as a CPU (Central Processing Unit) and a memory 32 serving as a storage device, as shown in FIG. 24. For example, a program for performing the method (video processing method) in the embodiment may be stored in the memory 32, and each function may be realized by the processor 31 executing the program stored in the memory 32.

[0094] These programs include instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The programs may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0095] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

[0096] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A video processing system comprising: image quality control means for controlling the image quality of a gaze area including a gaze target in an input video; recognition means for performing recognition processing on an object included in the video for which the image quality of the gaze area has been controlled; and extraction means for extracting the gaze target based on uncertainty of a recognition result of the recognition processing. (Supplementary Note 2) The video processing system according to Supplementary Note 1, wherein the recognition processing includes processing for recognizing the behavior of the object, and the extraction means extracts the gaze target based on variability in the recognition result of the behavior. (Supplementary Note 3) The video processing system according to Supplementary Note 2, wherein the extraction means determines the object whose behavior has been recognized as the gaze target when variability in the recognition result of the behavior is not within a predetermined range. (Supplementary Note 4) The video processing system according to Supplementary Note 2 or 3, wherein the recognition means includes a plurality of behavior predictors trained with different learning data, and the extraction means extracts the gaze target based on a variance in the recognition results of a plurality of behaviors output from the plurality of behavior predictors. (Supplementary Note 5) The video processing system according to Supplementary Note 2 or 3, wherein the recognition means includes a behavior predictor, and the extraction means extracts the gaze target based on a variance in the recognition results of a plurality of behaviors output by the behavior predictors when recognizing the behavior multiple times. (Supplementary Note 6) The video processing system according to any one of Supplementary Notes 2 to 5, wherein the extraction means extracts the gaze target based on a variance in behavior labels included in the recognition results of the behaviors or a variance in the scores of the behavior labels. (Supplementary Note 7) The video processing system according to Supplementary Note 5, wherein the behavior predictor outputs the recognition results of the plurality of behaviors by inactivating different nodes of a neural network for each recognition of the behavior. (Supplementary Note 8) A video processing method comprising: controlling the image quality of a gaze area including a gaze target in an input video; performing a recognition process on an object included in the video in which the image quality of the gaze area has been controlled; and extracting the gaze target based on uncertainty of a recognition result of the recognition process.(Supplementary Note 9) The video processing method according to Supplementary Note 8, wherein the recognition processing includes processing for recognizing the behavior of the object, and wherein the gaze target is extracted based on a variance in the recognition results of the behavior. (Supplementary Note 10) The video processing method according to Supplementary Note 9, wherein, in extracting the gaze target, if the variance in the recognition results of the behavior is not within a predetermined range, the object whose behavior has been recognized is determined to be the gaze target. (Supplementary Note 11) The video processing method according to Supplementary Note 9 or 10, wherein the behavior of the object is recognized by a plurality of behavior predictors that have learned different learning data, and the gaze target is extracted based on a variance in the recognition results of a plurality of behaviors output from the plurality of behavior predictors. (Supplementary Note 12) The video processing method according to Supplementary Note 9 or 10, wherein the behavior of the object is recognized by a behavior predictor, and the gaze target is extracted based on a variance in the recognition results of a plurality of behaviors output by the behavior predictors as a result of the behavior predictors recognizing the behavior multiple times. (Supplementary Note 13) The video processing method according to any one of Supplementary Notes 9 to 12, wherein the gaze target is extracted based on a variance in behavior labels included in the behavior recognition results or a variance in the scores of the behavior labels. (Supplementary Note 14) The video processing method according to Supplementary Note 12, wherein the behavior predictor outputs the recognition results of the multiple behaviors by inactivating different nodes of a neural network for each behavior recognition. (Supplementary Note 15) A video processing device comprising: image quality control means for controlling the image quality of a gaze area including the gaze target in an input video; recognition means for performing recognition processing on an object included in the video in which the image quality of the gaze area has been controlled; and extraction means for extracting the gaze target based on uncertainty of the recognition result of the recognition processing. (Supplementary Note 16) The video processing device according to Supplementary Note 15, wherein the recognition processing includes processing for recognizing the behavior of the object, and the extraction means extracts the gaze target based on a variance in the behavior recognition results. (Supplementary Note 17) The image processing device according to Supplementary Note 16, wherein the extraction means determines an object that recognizes the behavior as the gaze target when a variation in the behavior recognition result is not included in a predetermined range.(Supplementary Note 18) The video processing device according to Supplementary Note 16 or 17, wherein the recognition means includes a plurality of behavior predictors that have learned different learning data, and the extraction means extracts the gaze target based on a variance in recognition results of a plurality of behaviors output from the plurality of behavior predictors. (Supplementary Note 19) The video processing device according to Supplementary Note 16 or 17, wherein the recognition means includes a behavior predictor, and the extraction means extracts the gaze target based on a variance in recognition results of a plurality of behaviors output by the behavior predictor as a result of the behavior predictor recognizing the behavior a plurality of times. (Supplementary Note 20) The video processing device according to Supplementary Note 19, wherein the behavior predictor outputs recognition results of the plurality of behaviors by inactivating different nodes of a neural network for each recognition of the behavior.

[0097] 1 Remote monitoring system 10 Video processing system 11 Image quality control unit 12 Recognition unit 13 Extraction unit 20 Video processing device 30 Computer 31 Processor 32 Memory 100 Terminal 101 Camera 102 Compression efficiency optimization function 103 Video transmission function 110 Video acquisition unit 120 Detection unit 130 Image quality change determination unit 131 First determination unit 132 Second determination unit 140 Compression efficiency determination unit 150 Terminal communication unit 200 Center server 201 Video recognition function 202 Alert generation function 203 GUI drawing function 204 Screen display function 210 Center communication unit 220 Decoder 230 Behavior recognition unit 231 Object detection unit 232 Tracking unit 233 Behavior predictor 234 Behavior determination unit 235 Dropout setting unit 236 Variation calculation unit 240 Analysis information storage unit 250 Gaze target analysis unit 300 Base station 400 MEC 401 Compression bit rate control function 402 Terminal control function PM, PM1 to PM3 Predictor

Claims

1. an image quality control means for controlling the image quality of a gaze area including a gaze target in an input image; A recognition means for performing a recognition process for an object included in the image in which the image quality of the gaze area is controlled; an extraction means for extracting the gaze target based on uncertainty of a recognition result of the recognition processing; An image processing system comprising:

2. The recognition process includes a process of recognizing a behavior of the object, The extraction means extracts the gaze target based on a variance in the recognition result of the behavior. The video processing system according to claim 1 .

3. When a variation in the recognition result of the behavior does not fall within a predetermined range, the extraction means determines the object that recognizes the behavior as the gaze target. The video processing system according to claim 2 .

4. The recognition means includes a plurality of behavior predictors trained with different training data, The extraction means extracts the gaze target based on a variance in recognition results of a plurality of behaviors output from the plurality of behavior predictors.

4. The image processing system according to claim 2.

5. The recognition means includes a behavior predictor; The extraction means extracts the gaze target based on a variance in a plurality of behavior recognition results output by the behavior predictor performing behavior recognition a plurality of times.

4. The image processing system according to claim 2.

6. the extraction means extracts the gaze target based on a variation in an action label included in a recognition result of the action or a variation in a score of the action label; 4. The image processing system according to claim 2.

7. the behavior predictor outputs a recognition result of the plurality of behaviors by deactivating a different node of a neural network for each recognition of the behavior. The video processing system according to claim 5 .

8. Controlling the image quality of a gaze area including a gaze target in an input image; performing a recognition process for an object included in the image in which the image quality of the gaze area has been controlled; extracting the gaze target based on uncertainty of a recognition result of the recognition processing; Image processing method.

9. an image quality control means for controlling the image quality of a gaze area including a gaze target in an input image; A recognition means for performing a recognition process for an object included in the image in which the image quality of the gaze area is controlled; an extraction means for extracting the gaze target based on uncertainty of a recognition result of the recognition processing; The image processing device includes:

10. Controlling the image quality of a gaze area including a gaze target in an input image, performing a recognition process for an object included in the image in which the image quality of the gaze area has been controlled; extracting the gaze target based on uncertainty of a recognition result of the recognition processing; An image processing program that causes a computer to execute the processing.