Image processing device, image processing method, and image processing system

Abstract

To provide an image processing device that provides highly accurate activity detection means for wide-angle video from a first-person viewpoint.SOLUTION: An image processing device includes: an input unit that receives a wide-angle video including a user hand as an input video; a determination unit that determines a target area including the hand in the input video; an extraction unit that extracts a partial image corresponding to the target area from the input video; and a detection unit that detects activity being performed by a user by analyzing the partial image using a predetermined machine learning method and produces an activity detection result for the detected activity.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing system. [Background technology]

[0002] In recent years, amid a rapid increase in the amount of video content, including photographs, videos, and images, there has been a growing demand for technology to apply image processing technology to maintenance work to improve the accuracy of work in order to promote social innovation using IoT. In particular, there is an increasing demand for technology that utilizes video content to accurately determine the work performed by users such as workers. In the technical field of video analysis, detecting the behavior or situation of a specific object is called "activity detection."

[0003] There have been several proposals for such activity detection. For example, U.S. Patent Application Publication No. 2017 / 0220854 (Patent Document 1) describes a technology in which "a multimodal sensing system includes various devices that work together to automatically classify actions. A video camera captures a series of digital images. At least one other sensor device acquires other sensor data (e.g., motion data). The system extracts video features from the digital images and associates each extracted image feature with a time period. The system also extracts other features from the other sensor data and associates each other extracted feature with a time period. The system combines a group of extracted video features with a group of extracted other features to create a combined feature representation that corresponds to a predetermined time period. The system then analyzes the combined feature representation to identify classes, accesses a data store of classes and actions, identifies actions associated with the classes, and stores the identified actions in a memory device."

[0004] In addition, Lenjani et al., "Automated Building Image Extraction from 360 Panoramas for Post-Disaster evaluation" (Non-Patent Document 1) states, "After a disaster, a team of structural engineers collects a large number of images from damaged buildings to acquire new knowledge and learn lessons from the disaster. However, the collected images are often taken without sufficient spatial context. When the damage is severe, it may be difficult to distinguish buildings. In order to accurately identify the cause of the building's defects or actual loss, it is necessary to access images of the pre-disaster state of these buildings. Here, to address this problem, we propose a method to automatically extract pre-disaster building images from 360° panoramic images (panoramas). By providing as input a geotagged image collected near the target building, a panorama close to the location of the input image is automatically downloaded via a street view service (such as Google or Bing in the United States). By calculating the geometric relationship between the panorama and the target building, the optimal projection direction for each panorama is identified and a high-quality 2D image of the building is generated. A region-based convolutional neural network is utilized to recognize the building in these 2D images. Several panoramas are used so that the detected building images display various perspectives of the building." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] US Patent Application Publication No. 2017 / 0220854 [Non-patent literature]

[0006] [Non-Patent Document 1] Lenjani et al., “Automated Building Image Extraction from 360 Panoramas for Post-Disaster evaluation” Summary of the Invention [Problem to be solved by the invention]

[0007] Patent document 1 describes a means for detecting activities taking place in a video by combining features extracted from a first-person perspective video captured by a wearable camera with features extracted from sensor data captured by other sensors such as an acceleration sensor, and processing the combined features using a neural network. However, in the method of Patent Document 1, the first-person viewpoint image captured by the wearable camera is an image with a standard angle of view, so the range of the activity detection process is limited to a narrow field of view of the image, and the activity to be detected may not be included in the shooting range of the wearable camera. In this case, even if the image thus captured is processed by a neural network trained to detect activity, the accuracy of activity detection will be low.

[0008] Non-Patent Document 1 describes a means for assessing the scale of damage caused by a disaster by correlating and analyzing 360-degree panoramic video showing the state of a building before the disaster with video showing the state of the building after the disaster. However, the means of Non-Patent Document 1 extracts an area of ​​interest from a 360-degree panoramic video based on data from other sensors such as GPS, and does not assume cases where data from other sensors such as GPS is unavailable, or where it is desired to detect activity related to a small object that cannot be identified by GPS data, etc. Therefore, the means of Non-Patent Document 1 is not suitable for detecting work (such as tightening a screw or hammering a nail) performed by a worker in maintenance work, for example.

[0009] Therefore, the present disclosure aims to provide an image processing means that detects a user's hand in a wide-angle first-person perspective image captured by a wide-angle wearable camera worn by the user, extracts an image of a target area based on the position of the detected user's hand, and performs activity detection on the extracted image, thereby enabling highly accurate activity detection in the wide-angle first-person perspective image. [Means for solving the problem]

[0010] In order to solve the above problems, one representative image processing device of the present disclosure includes an input unit that accepts a wide-angle image including a user's hand as an input image, a determination unit that determines a target area including the hand in the input image, an extraction unit that extracts a partial image corresponding to the target area from the input image, and a detection unit that detects an activity being performed by the user by analyzing the partial image using a predetermined machine learning method, and generates an activity detection result related to the detected activity. Effect of the Invention

[0011] According to the present disclosure, it is possible to provide an image processing means that detects a user's hands in wide-angle first-person perspective image captured by a wide-angle wearable camera worn by the user, extracts an image of a target area based on the detected position of the user's hands, and performs activity detection on the extracted image, thereby enabling highly accurate activity detection in wide-angle first-person perspective image. Other objects, configurations and effects will become apparent from the following description of the preferred embodiment of the invention. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates a computer system for implementing an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of a configuration of an image processing system according to the first embodiment of the present disclosure. [Diagram 3]FIG. 3 is a block diagram illustrating an overall data flow in the image processing device according to the first embodiment of the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating a process flow for extracting a partial image according to the first embodiment of the present disclosure. [Diagram 5] FIG. 5 is a block diagram illustrating an example of a flow of a target region determination process according to the first embodiment of the present disclosure. [Figure 6] FIG. 6 is a flowchart illustrating an example of a flow of a hand detection process according to the first embodiment of the present disclosure. [Figure 7] FIG. 7 is a block diagram illustrating a training process of a neural network trained to identify hand positions in a wide-angle image according to the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a flowchart illustrating an example of a flow of a hand detection process according to the first embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram illustrating an example of a learning data table that stores learning data used in the training process in the neural network according to the first embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating an example of a prediction result table for storing information on the prediction results by the neural network according to the first embodiment of the present disclosure. [Figure 11] FIG. 11 is a diagram illustrating an example of a hand position information table storing hand position information generated by the hand detection process according to the first embodiment of the present disclosure. [Figure 12] FIG. 12 is a diagram illustrating an example of a tracking result table storing information of a tracking result generated by a hand tracking process according to the first embodiment of the present disclosure. [Figure 13] FIG. 13 is a flowchart illustrating an example of the flow of the target region determination process according to the first embodiment of the present disclosure. [Figure 14] FIG. 14 is a diagram illustrating an example of a target area information table storing information on a target area determined by a target area determination process according to the first embodiment of the present disclosure. [Figure 15] FIG. 15 is a flowchart illustrating a flow of the partial image extraction process according to the first embodiment of the present disclosure. [Figure 16] FIG. 16 is a diagram illustrating an example of a partial image management table that stores information on partial images extracted by the partial image extraction process according to the first embodiment of the present disclosure. [Figure 17] FIG. 17 is a block diagram illustrating a flow of a target region determination process according to the second embodiment of the present disclosure. [Figure 18] FIG. 18 is a flowchart illustrating a flow of a right / left hand discrimination process according to the second embodiment of the present disclosure. [Figure 19] FIG. 19 is a diagram illustrating an example of a right / left hand position information table storing the position information of the right / left hand of a user generated by the right / left hand identification process according to the second embodiment of the present disclosure. [Figure 20] FIG. 20 is a flowchart illustrating an example of the flow of a partial image extraction process according to the second embodiment of the present disclosure. [Figure 21] FIG. 21 is a diagram illustrating an example of a partial image management table that stores information on partial images extracted by the partial image extraction process according to the second embodiment of the present disclosure. [Figure 22] FIG. 22 is a diagram illustrating a configuration of a user interface according to the second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] As described above, an embodiment of the present disclosure relates to an image processing means that enables highly accurate activity detection for wide-angle first-person viewpoint images. In the present disclosure, a wide-angle video from a first-person perspective is acquired as an input video. The input video may be acquired, for example, by a wide-angle wearable camera worn by a user. The input video is composed of a number of image frames (hereinafter referred to as "frames"). In each frame of the input video, a user's hands (one or both hands) are detected, and a target region based on the position of the detected user's hands is determined. Then, a partial image corresponding to the target region is extracted from each frame and analyzed by a neural network trained for activity detection. Additionally, in some embodiments of the present disclosure, the user's right and left hands are each separately identified, making it possible to determine whether an activity of interest is being performed by the right hand, the left hand, or both hands. Furthermore, by acquiring handedness information indicating the handedness of the user, the handedness of the user in each frame can be identified, which makes it possible to detect activity centered on the handedness of the user.

[0014] In the embodiment of the present disclosure, a wide-angle image from a first-person perspective is used as the input image, thereby avoiding the problem that the activity to be detected is not included in the shooting range of the wearable camera. In addition, in the embodiment of the present disclosure, the user's hand is detected, and activity detection is performed on a partial image of the target area including the detected user's hand. This makes it possible to perform high-precision activity detection even when data from other sensors such as GPS is not available or when it is desired to detect an activity related to a small object that cannot be identified by GPS data.

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to the embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals.

[0016] First, referring to Fig. 1, a computer system 100 for implementing the embodiments of the present disclosure will be described. The mechanisms and devices of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 100 include one or more processors 102, memory 104, terminal interface 112, storage interface 113, I / O (input / output) device interface 114, and network interface 115. These components may be interconnected via a memory bus 106, an I / O bus 108, a bus interface unit 109, and an I / O bus interface unit 110.

[0017] Computer system 100 may include one or more general purpose programmable central processing units (CPUs) 102A and 102B, collectively referred to as processors 102. In some embodiments, computer system 100 may include multiple processors, and in other embodiments, computer system 100 may be a single CPU system. Each processor 102 executes instructions stored in memory 104 and may include an on-board cache.

[0018] In one embodiment, memory 104 may include random access semiconductor memory, storage devices, or storage media (either volatile or non-volatile) for storing data and programs. Memory 104 may store all or a portion of the programs, modules, and data structures that implement the functions described herein. For example, memory 104 may store an image processing application 150. In one embodiment, image processing application 150 may include instructions or descriptions that execute on processor 102 the functions described below.

[0019] In some embodiments, image processing application 150 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, image processing application 150 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with bus interface unit 109, processor 102, or other hardware of computer system 100.

[0020] Computer system 100 may include a bus interface unit 109 that provides communication between processor 102, memory 104, display system 124, and I / O bus interface unit 110. I / O bus interface unit 110 may couple to an I / O bus 108 for transferring data to and from various I / O units. I / O bus interface unit 110 may communicate via I / O bus 108 with multiple I / O interface units 112, 113, 114, and 115, also known as I / O processors (IOPs) or I / O adapters (IOAs).

[0021] Display system 124 may include a display controller, a display memory, or both. The display controller may provide video, audio, or both data to display device 126. Computer system 100 may also include one or more sensors or other devices configured to collect data and provide the data to processor 102.

[0022] For example, computer system 100 may include biometric sensors to collect heart rate data, stress level data, etc., environmental sensors to collect humidity data, temperature data, pressure data, etc., and motion sensors to collect acceleration data, movement data, etc. Other types of sensors may also be used. Display system 124 may be connected to a display device 126, such as a standalone display screen, a television, a tablet, or a handheld device.

[0023] The I / O interface unit provides the ability to communicate with various storage or I / O devices. For example, the terminal interface unit 112 may be attached to user I / O devices 116, such as user output devices, such as a video display, a television with speakers, and user input devices, such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing device. A user may use a user interface to enter input data or instructions to the user I / O devices 116 and the computer system 100, and receive output data from the computer system 100, by manipulating the user input devices. The user interface may be displayed on a display, played through speakers, or printed via a printer, for example, via the user I / O devices 116.

[0024] Storage interface 113 may be attached to one or more disk drives or direct access storage device 117 (usually a magnetic disk drive storage device, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In one embodiment, storage device 117 may be implemented as any secondary storage device. Contents of memory 104 may be stored in storage device 117 and retrieved from storage device 117 as needed. I / O device interface 114 may provide an interface to other I / O devices such as printers, fax machines, etc. Network interface 115 may provide a communications path to allow computer system 100 and other devices to communicate with each other. This communications path may be, for example, network 130.

[0025] In some embodiments, computer system 100 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 100 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

[0026] <Example 1> Next, a first embodiment of the present disclosure will be described with reference to FIGS.

[0027] 2 is a diagram illustrating an example of a configuration of an image processing system 200 according to a first embodiment of the present disclosure. The image processing system 200 is a system for detecting an activity in a wide-angle image from a first-person viewpoint. 2, an image processing system 200 according to the first embodiment of the present disclosure includes a client terminal 210, an image capturing device 220 worn by a user 230, a communication network 240, and an image processing device 250. The client terminal 210, the image capturing device 220, and the image processing device 250 are connected to each other via the communication network 240. The communication network 240 here may include, for example, the Internet, a LAN (Local Area Network), a MAN (Metropolitan Area Network), a WAN (Wide Area Network), etc.

[0028] The client terminal 210 is a terminal used by a requester of activity detection. The client terminal 210 may select a wide-angle image to be input, a frame to be processed, information to be output, and the like, for example, using a user interface (see FIG. 22) described later, and then transmit an activity detection request to the image processing device 250 via the communication network 240, which requests activity detection processing. The client terminal 210 may be a mobile terminal such as a smartphone or a tablet, or may be a fixed terminal such as a desktop computer. The client terminal 210 may also be equipped with an input function such as a touch screen, a keyboard, a mouse, or a voice recognition function for receiving input from a user.

[0029] The user 230 is a user who performs an activity that is the target of activity detection, and may be, for example, a worker performing maintenance work, etc. Here, the "activity" may be an action performed by the user 230 (throwing a ball, tightening a screw, etc.), an action performed on the user 230 (being bitten by a dog, etc.), or any event occurring in the same spatiotemporal region as the user 230. 2, a user 230 may wear an image capture device 220. Additionally, in some embodiments, the user 230 may wear other sensors, such as an accelerometer or a gyroscope.

[0030] The photographing device 220 is a device for acquiring an input image to be the target of activity detection. The photographing device 220 may be, for example, a wide-angle camera (e.g., GoPro Max, etc.) worn by the user 230, and may be configured to continuously capture wide-angle images (e.g., 360-degree images) from a first-person perspective while the user 230 is moving. The wide-angle images here refer to images captured by a lens with a focal length of 35 mm or less, for example. The image capturing device 220 may be attached to a position where it can easily capture an image of the user's hands, such as the head or chest of the user 230. The wide-angle image from the first-person perspective captured by the image capturing device 220 is transmitted to the image processing device 250 via the communication network 240.

[0031] The image processing device 250 is a device for detecting an activity in a wide-angle image from a first-person perspective. The image processing device 250 may be, for example, a remote server that is geographically separated from the user 230 and the client terminal 210, and may be configured to transmit an activity detection result for a wide-angle image acquired from the image capture device 220 to the client terminal 210 as a service via the communication network 240. However, the present invention is not limited to this. For example, each functional unit of the image processing device 250 may be implemented as a software application executed on the client terminal 210. 2, the image processing device 250 includes an input unit 252, a determination unit 254, an extraction unit 256, a detection unit 258, an output unit 260, and a storage unit 262. However, the image processing device 250 is not limited to this configuration, and may include other functional units as described later.

[0032] The input unit 252 is a functional unit that receives a wide-angle image acquired by the image capturing device 220 as an input image. This wide-angle image may be, for example, a wide-angle image from a first-person perspective including the hands (one hand or both hands) of the user 230. The input unit 252 may continuously receive the wide-angle image from the image capturing device 220 in real time, or may receive the image periodically. The input unit 252 may also receive sensor data acquired by other sensors such as an acceleration sensor or a gyroscope. The determination unit 254 is a functional unit that determines a target region including the hand of the user 230 in the input video acquired by the input unit 252. The determination unit 254 may be, for example, a neural network trained to identify the position of the hand in the video. The extraction unit 256 is a functional unit that extracts a partial image corresponding to the target region determined by the determination unit 254 from the input video. The detection unit 258 is a functional unit that detects an activity being performed by the user 230 by analyzing the partial image extracted by the extraction unit 256 using a predetermined machine learning method, and generates an activity detection result regarding the detected activity. The output unit 260 is a functional unit that outputs the activity detection result generated by the detection unit 258 to the client terminal 210 and the storage unit 262 . The storage unit 262 is a functional unit for storing the activity detection results output by the output unit 260. The storage unit 262 may be any storage medium, such as a flash memory or a hard disk drive.

[0033] According to the image processing system 200 described above, activity detection from a wide-angle first-person viewpoint image with high accuracy becomes possible.

[0034] Next, with reference to FIG. 3, an overall data flow in the image processing device according to the first embodiment of the present disclosure will be described. FIG. 3 is a block diagram showing the overall data flow in the image processing device 250 according to the first embodiment of the present disclosure.

[0035] First, the input unit 252 receives a wide-angle image 302 acquired by the above-mentioned image capturing device as an input image. As described above, this wide-angle image 302 is a wide-angle image from a first-person perspective that includes, for example, the user's hands (one hand or both hands). This input image is input to the determination unit 254. In the present disclosure, it is assumed that the activity to be detected is performed around the user's hand. Therefore, the determination unit 254 determines a target area including the user's hand in the input image from the input unit 252. The activity to be detected should be included in this target area including the user's hand. Next, the extraction unit 256 extracts a partial image 310 corresponding to the target region determined by the determination unit 254. This partial image 310 is input to the feature extraction unit 320. The feature extraction unit 320 here may be a hand-crafted feature extraction method such as SIFT (Space Invariant Feature Transform), or a neural network such as ResNet or VGG (Visual Geometry Group) pre-trained by open source training data (Imagenet, Kinetics) or custom training data. The feature extraction unit 320 extracts features 324 of the partial image 310.

[0036] As described above, the image processing device 250 according to the embodiment of the present disclosure may use sensor data acquired by other sensors, such as an acceleration sensor or a gyroscope, for activity detection in addition to the wide-angle image 302. In this case, the input unit 252 receives sensor data 304 acquired by other sensors, such as an acceleration sensor or a gyroscope, and then inputs the received sensor data 304 to the feature extraction unit 320. The feature extractor 320 then extracts features 322 from the sensor data 304 .

[0037] Next, the feature combination unit 325 combines the partial image feature 324 and the sensor data feature 322 in a time-series correspondence. Here, the feature combination unit 325 may use a feature concatenation method as a means for combining features. In this feature concatenation method, the partial image feature 324 is associated with and combined with the feature with the closest timestamp among the sensor data features 322. It should be noted that in the absence of sensor data 304 , the feature combination unit 325 inputs the partial image features 324 directly to the machine learning unit 340 .

[0038] The features combined by the feature combiner 325 are input to a machine learning unit 340, which may include neural networks (ResNet or VGG) or deep learning algorithms (Support Vector Machines) trained to detect activity in videos based on open source or custom training data, where custom training data refers to data that is specially created for the purpose of training. As a result of the processing by the machine learning unit 340, an activity (i.e., an activity performed by a user's hand movement) in the wide-angle image 302 is detected, and an activity detection result 350 indicating the activity is output. This activity detection result 350 may include, for example, a label indicating the type of activity and the time (start time and end time) at which the activity is performed in the wide-angle image. Furthermore, this activity detection result 350 may be stored in a storage unit of the image processing device, or may be transmitted to a client terminal.

[0039] According to the image processing device 250 described above, activity detection from a wide-angle first-person viewpoint image with high accuracy becomes possible.

[0040] Next, a process flow for extracting a partial image according to the first embodiment of the present disclosure will be described with reference to FIG.

[0041] FIG. 4 is a block diagram illustrating a process flow for extracting a partial image according to the first embodiment of the present disclosure.

[0042] First, the input unit 252 receives the wide-angle image 302 acquired by the above-mentioned imaging device as the input image 405. As described above, the wide-angle image 302 is a wide-angle image from a first-person perspective, including, for example, the user's hands (one hand or both hands). Since the input image 405 is a wide-angle image, it includes a lot of visual information related to the user's surrounding environment in addition to the user's hands. If the input image 405 is directly input to the detection unit and activity detection is performed, the detection unit will be confused by visual information unrelated to the target activity, and the detection accuracy will be limited. On the other hand, by determining a target area centered on visual information useful for activity detection, such as the area around the user's hands, and performing activity detection on a partial image extracted based on the target area, it is possible to perform activity detection with high accuracy.

[0043] Next, determination unit 254 inputs input video 405 and determines a target region including the user's hand in input video 405. The target region here means a spatial region including the activity to be detected in input video 405. In general, the activity to be detected is performed around the user's hand, so the target region including the hand should include the activity to be detected. The determination unit 254 inputs the coordinates that define the determined target region to the extraction unit 256 . The details of the determination of the target region by the determination unit 254 will be described later (see FIG. 13), and therefore will not be described here.

[0044] Next, based on the coordinates of the target region input from the determination unit 254, the extraction unit 256 extracts a partial image 310 corresponding to the target region from the input video 405. As will be described later, here, the extraction unit 256 may extract one partial image 310 from the input video 405 or may extract multiple partial images 310 from the input video 405 based on the spatial position of the user's hand, handedness information related to the user's handedness, or the like. The extraction of the partial image 310 by the extractor 256 will be described in detail later (see FIG. 15), and therefore will not be described here.

[0045] 4, it is possible to determine a target area centered on visual information useful for activity detection, such as the area around a user's hand, and extract a partial image 310 corresponding to the target area from the input video 405. By performing activity detection on the partial image 310 extracted in this manner, it is possible to perform activity detection with high accuracy.

[0046] Next, a process flow for determining a target region according to the first embodiment of the present disclosure will be described with reference to FIG.

[0047] FIG. 5 is a block diagram illustrating an example of a process flow for determining a target region according to the first embodiment of the present disclosure. As described above, the present disclosure is based on the premise that the activity to be detected is taking place around the user's hands, and therefore the activity to be detected should be included in the target area including the user's hands.

[0048] First, the determination unit inputs the input video 405, and then executes a process of detecting a hand for a specific frame (e.g., the first frame) in the input video 405 (505). Here, the determination unit may use a neural network (e.g., VGG16-SSD or YOLOv3) trained to detect hands based on open source learning data such as Egohands or custom learning data in order to detect the user's hand in the input video 405. This neural network may be trained to detect not only bare hands, but also hands holding objects such as tools and hands wearing gloves. The neural network estimates and outputs a region 510 including each hand in the input video 405. Here, the neural network may define a bounding box as the region 510 including the hand, and output coordinates that define the position of the bounding box.

[0049] Next, the determination unit performs a process of tracking the hand (515). Here, the determination unit identifies the hand in each frame constituting the input video 405 based on the region including the hand 510 estimated for a particular frame. Here, by performing the process of tracking the hand, the determination unit can determine the region including the user's hand (e.g., a bounding box) for each frame constituting the input video 405. Here, as the process of tracking the hand, the determination unit may use a so-called Kalman filter or a neural network trained to track the hand based on open source learning data or custom learning data. By using this process of tracking the hand, for example, when the hand of another person appears in the input image 405 in addition to the hand of the user wearing the camera device, it becomes possible to distinguish the hand of the user wearing the camera device from the hand of the other person. In addition, in order to facilitate tracking of the hand, the image processing system instructs the user to hold the hand straight in front of the camera device when the camera device starts capturing a wide-angle image. This makes it easier to distinguish the hand of the user wearing the camera device from the hand of the other person. In addition, if the hand tracking process determines that the hand of the user wearing the photographing device is no longer within the photographing range of the photographing device, the determination unit may provide the user with a notification to return the hand to the photographing range of the photographing device, for example via a user interface described later (see Figure 22).

[0050] Next, the determination unit performs a hand tracking process to identify the user's hands in each frame constituting the input video 405, and then determines 520 a target region 525 based on the position of the identified region including the hands (i.e., the coordinates of the bounding box). As described above, this target region 525 should include the activity to be detected.

[0051] According to the process described above with reference to FIG. 5, it is possible to determine a target area in input video 405 that includes the activity to be detected.

[0052] Next, an example of a flow of a hand detection process according to the first embodiment of the present disclosure will be described with reference to FIG.

[0053] As described above, in the embodiment of the present disclosure, the hand detection process 600 for estimating the position of the hand is performed on a wide-angle image such as a 360-degree image. However, the wide-angle image has a problem that it is difficult to detect an object compared to an image with a standard angle of view because the wide-angle image has radial distortion due to the camera lens and no clear edges. Therefore, the hand detection process 600 according to the embodiment of the present disclosure has been made in consideration of this problem, and can detect an object such as a user's hand with high accuracy even in a wide-angle image such as a 360-degree image that has distortion. 6 is a flowchart showing an example of the flow of a hand detection process 600 according to the first embodiment of the present disclosure. The hand detection process 600 is a process for detecting a hand in an input image that is a wide-angle image using a neural network trained to identify the position of a hand in a standard image that is not a wide-angle image, and is executed by a determination unit.

[0054] First, in step S610, the determination unit 100 judges whether a wide-angle image (F 1 , F 2 , ...F N ) is received as an input image. As described above, this input image may be, for example, a first-person perspective wide-angle image captured by the above-mentioned imaging device and received by an input unit of the image processing device.

[0055] Next, in step S620, the determination unit constructs a neural network D (first neural network) trained by images with a standard angle of view. Here, the determination unit may construct the neural network D by training a neural network such as VGG16-SSD or YOLOv3 using images with a standard angle of view as learning data. It should be noted that the standard angle of view here refers to an image captured by a lens with a focal length of, for example, 40 mm or more.

[0056] Next, in step S630, the determination unit declares a variable "n" as a frame index of a frame in the input video, and sets n to 1. This frame index n is a variable that specifies which frame is to be processed among multiple frames included in the input video.

[0057] Next, in step S640, the determination unit 100 determines whether the frame F n Determine whether the frame index n of the frame to be processed F is 1 (i.e., the first frame of the input video). n If the frame index n of the frame to be processed is 1, the process proceeds to step S650, and if the frame index n of the frame to be processed is not 1, the process proceeds to step S660.

[0058] As described above, in the present disclosure, the image processing system instructs the user to hold his / her hand straight in front of the image capture device when the image capture device starts capturing a wide-angle image. This causes the user's hand to be positioned in frame F 1 Therefore, the frame F to be processed n is the first frame of the input video. 1 In this case (i.e., when the frame index n is 1), the user's hand can be detected with high accuracy even if a neural network D trained with a standard angle of view image, not a wide-angle image, is used.

[0059] On the other hand, the frame F n F 1 In other cases (i.e., when the frame index n is not 1), the user's hand is not necessarily clearly captured, and even if the entire frame is processed by the neural network D trained on the standard angle of view image, the user's hand may not be identified with high accuracy. Therefore, in the embodiment of the present disclosure, the frame F to be processed is n F 1 For frames other than , the previous frame F n-1 The image of the region corresponding to the hand region estimated for the frame F to be processed is nThis is because the displacement (i.e., movement) of the user's hand between two consecutive frames is relatively small, so the frame F n The hand position in and the previous frame F n-1 This is because the hand position in the

[0060] Then, the extracted image can be processed by an image processing method such as OpenCV to eliminate distortion and convert the image into an image with a standard angle of view. By processing the pseudo-standard images (first pseudo-standard image, second pseudo-standard image, etc.) converted into images with a standard angle of view in this way using a neural network D trained by the images with the standard angle of view, the coordinates of the area including the user's hand (e.g., a bounding box) can be estimated with high accuracy.

[0061] Next, in step S650, the determination unit 100 determines whether the frame F n If the frame index n of is 1 (i.e., the first frame of the input video), then the frame F 1 By inputting the whole of the frame F into the neural network D, 1 The position of the hand in the image is estimated, and the coordinates of the region (e.g., a bounding box) containing the hand are estimated. Then, the process proceeds to step S680.

[0062] Next, in step S660, the determination unit 100 determines whether the frame F n Frame F, the frame immediately before n-1 The image of the area corresponding to the hand area detected in the frame F to be processed is n Extract from. In this case, the determination unit determines whether the frame F n-1 For each hand in the frame F n In this manner, two images (e.g., a first image and a second image) corresponding to both hands of the user can be extracted.

[0063] Next, in step S670, the determination unit 100 determines whether the frame Fn The image (e.g., the first image and the second image) extracted from is processed by an image processing method such as OpenCV to eliminate distortion and generate pseudo-standard images (e.g., the first pseudo-standard image and the second pseudo-standard image) by converting the image into an image with a standard angle of view. After that, the determination unit processes the secondary standard image by a neural network D trained by the image with the standard angle of view, thereby estimating the coordinates of the area (e.g., the bounding box) including the user's hand.

[0064] Next, in step S680, the decision unit increments the frame index n by 1 (that is, n becomes n+1).

[0065] Next, in step S690, the determination unit 100 determines whether the frame F n It is determined whether the frame index n of the frame F to be processed is greater than the number N of frames contained in the input video. n If the frame index n of the frame F is greater than the number N of frames included in the input video, the process proceeds to step S695. n If the frame index n is less than or equal to the number N of frames contained in the input video, the process returns to step S640 and the subsequent steps are repeated for the next frame.

[0066] Next, in step S695, after all frames constituting the input video have been processed, the determination unit 102 determines, for each frame F, the area including the hand in that frame (for example, the bounding box {H 1 ,H 2 ...H N}) to output the coordinates.

[0067] According to the hand detection process 600 described above, for each frame constituting an input image, which is a wide-angle image, it is possible to estimate the position of the hand in that frame with high accuracy and output the coordinates of the area (e.g., a bounding box) containing the hand in the frame.

[0068] Next, a modified example of the hand detection process according to the embodiment of the present disclosure will be described with reference to FIGS. In the hand detection process 600 described above with reference to Fig. 6, a neural network trained to identify hand positions in a standard image, not a wide-angle image, is used to detect a hand in an input image, which is a wide-angle image, but the present disclosure is not limited thereto. For example, a neural network trained to identify hand positions in a wide-angle image may be used. Hereinafter, with reference to Figs. 7 and 8, a case where hand detection processing is performed using a neural network trained to identify the position of a hand in a wide-angle image will be described.

[0069] FIG. 7 is a block diagram illustrating a training process 700 for a neural network trained to identify hand positions in wide-angle images according to a first embodiment of the present disclosure.

[0070] First, images with a standard angle of view are acquired as the training data 705. The images with a standard angle of view that become the training data 705 may be, for example, an open source training data set or custom training data. In addition, ground truth bounding boxes 710 are acquired as annotation data indicating the correct positions of objects such as hands in the training data 705.

[0071] The image conversion unit 715 inputs the learning data 705, processes it using an image processing method such as OpenCV, and distorts it to convert it into a pseudo wide-angle image that resembles a wide-angle image. In addition, the ground truth bounding box 710 is corrected as necessary in accordance with the conversion of the learning data 705 image.

[0072] Next, the pseudo wide-angle image generated by image conversion unit 715 is input to object detection unit 720. This object detection unit 720 predicts the position of the hands in the pseudo wide-angle image based on object detection unit parameters 717. More specifically, object detection unit 720 outputs, for each hand in each pseudo wide-angle image, the coordinates of a predicted bounding box 730 that defines an area including the hand. Note that the object detector parameters 717 here may be initialized to random values ​​or to the parameters of a pre-trained neural network.

[0073] An error calculation unit 740 then calculates the error in the prediction of the object detection unit 720 by comparing the ground truth bounding box 710 (i.e. the ground truth bounding box corrected to the pseudo wide-angle image) with the predicted bounding box 730.

[0074] Next, the update unit 750 updates the parameters 717 of the object detection unit based on the error in the prediction of the object detection unit 720 calculated by the error calculation unit 740. Here, the update unit 750 may use a so-called backpropagation method to adjust the parameters 717 of the object detection unit so as to minimize the error in the prediction of the object detection unit 720.

[0075] The above training process 700 may be repeated until the error of the object detector 720 meets a predefined error criterion, resulting in a neural network that is trained to localize hands in wide-angle images.

[0076] 8 is a flowchart showing an example of the flow of a hand detection process 800 according to the first embodiment of the present disclosure. The hand detection process 800 is a process for detecting a hand in an input image, which is a wide-angle image, using a neural network trained to identify the position of a hand in a wide-angle image (i.e., the neural network obtained by the training process 700 described with reference to FIG. 7), and is executed by a determination unit.

[0077] First, in step S810, the determination unit 100 judges whether a wide-angle image (F 1 , F 2 , ...F N ) is received as an input image. As described above, this input image may be, for example, a first-person perspective wide-angle image captured by the above-mentioned imaging device and received by an input unit of the image processing device.

[0078] Next, in step S820, the determination unit constructs a neural network D1 (second neural network) trained by the wide-angle image. Here, the determination unit may construct the neural network D1 trained to identify the position of the hand in the wide-angle image by, for example, executing the training process 700 described with reference to FIG.

[0079] Next, in step S830, the determination unit declares a variable "n" as a frame index of a frame in the input video, and sets n to 1. This frame index n is a variable that specifies which frame is to be processed among multiple frames included in the input video.

[0080] Next, in step S840, the determination unit 100 determines whether the frame F n is input to the neural network D1 constructed in step S820, the frame F n The hand position in is estimated, and the coordinates of the region (e.g., a bounding box) containing the hand are estimated.

[0081] Next, in step S850, the decision unit increments the frame index n by 1 (that is, n becomes n+1).

[0082] Next, in step S860, the determination unit 100 determines whether the frame F n Determine whether the frame index n of the frame F to be processed is equal to the number of frames N contained in the input video. n If the frame index n of the frame to be processed is equal to the number N of frames included in the input video, the process proceeds to step S870. On the other hand, if the frame index n of the frame to be processed is not equal to the number N of frames included in the input video (e.g., n is smaller than N), the process returns to step S840 and the subsequent steps are repeated for the next frame.

[0083] Next, in step S870, after all frames constituting the input video have been processed, the determination unit n For each frame, we estimate the region that contains the hand (e.g., the bounding box {H 1 ,H 2 ,...H N})'s coordinates.

[0084] According to the hand detection process 800 described above, for each frame constituting the input image, which is a wide-angle image, it is possible to estimate the position of the hand in that frame with high accuracy and output the coordinates of the area (e.g., a bounding box) containing the hand in the frame. Furthermore, in hand detection process 800, all frames are input into a neural network in their entirety, and therefore, compared to hand detection process 600 described with reference to FIG. 7, there is no need to determine, extract, or process images of areas including hands, making it possible to reduce the processing load and processing time.

[0085] Next, with reference to FIG. 9, a data configuration of a learning data table that stores learning data used in the neural network training process according to the first embodiment of the present disclosure will be described.

[0086] 9 is a diagram illustrating an example of a training data table 900 for storing training data used in the neural network training process according to the first embodiment of the present disclosure. The training data stored in the training data table 900 is, for example, data used in the training process 700 described with reference to FIG.

[0087] As shown in FIG. 9, the training data table 900 includes training images 901 and ground truth bounding boxes 902. In the sequence of training images 901, an identifier T 1 ~T M For example, if there are M training images, the first training image is T 1 and the last training image is T M It is.

[0088] Additionally, the column Ground Truth Bounding Box 902 stores information about the ground truth bounding boxes that define the correct region of the hand in each training image. Each ground truth bounding box corresponds to a different hand in the training image (i.e., if a training image contains two hands, there will be two ground truth bounding boxes for that training image).

[0089] Each ground truth bounding box is m、k Here, m denotes the training image number, and k denotes the ground truth bounding box number (i.e., B 3、1 is the third training image T 3 (This is the first ground truth bounding box in

[0090] Each ground truth bounding box is denoted by ((x 1 ,y 1 ),(x 2 ,y 2),c), where (x 1 ,y 1 ) are the coordinates of the upper left corner of the bounding box, and (x 2 ,y 2 ) indicates the coordinates of the bottom right corner of the bounding box, and c indicates the class of the object (e.g., "hand").

[0091] As shown in the training data table 900 described above, by training a neural network using training data that associates wide-angle training images with ground truth bounding boxes that indicate the correct area of ​​the hand in each training image, it is possible to detect the area of ​​the hand in a wide-angle image with high accuracy.

[0092] Next, with reference to FIG. 10, a data configuration of a prediction result table for storing information on the prediction results by the neural network according to the first embodiment of the present disclosure will be described.

[0093] 10 is a diagram illustrating an example of a prediction result table 1000 for storing information on prediction results by a neural network according to the first embodiment of the present disclosure. The information on prediction results stored in the prediction result table 1000 is information generated by an object detection unit in the training process described with reference to FIG.

[0094] As shown in FIG. 10, a prediction result table 1000 includes a training image 1001 and a predicted bounding box 1002 . In the sequence of training images 1001, an identifier T 1 ~T M For example, if there are M training images, the first training image is T 1 and the last training image is T M It is. Additionally, the column of predicted bounding boxes 1002 stores information about predicted bounding boxes that define the regions of the hands predicted by the object detection unit. Each predicted bounding box corresponds to a different hand in the training image (i.e., if a training image contains two hands, there are two predicted bounding boxes for that training image).

[0095] Each predicted bounding box is m、k Here, m denotes the number of the training image, and k denotes the number of the predicted bounding box (i.e., P 2、2 is the second training image T 2 (This is the second predicted bounding box in

[0096] Each predicted bounding box is denoted by ((x 1 ',y 1 '),(x 2 ',y 2 The information is specified in the format of (x '),c',p). 1 ',y 1 ') indicates the coordinates of the upper left corner of the bounding box, and (x 2 ',y 2 Let a') denote the coordinates of the bottom right corner of the bounding box, c' denote the class of the object (e.g., "hand"), and p denote the confidence of the bounding box.

[0097] As shown in the prediction result table 1000 described above, for each hand in a specific learning image (e.g., a pseudo wide-angle image converted from an image with a standard angle of view), information on a predicted bounding box that defines an area including the hand can be managed.

[0098] Next, a data configuration of a hand position information table that stores hand position information generated by the hand detection process according to the first embodiment of the present disclosure will be described with reference to FIG.

[0099] 11 is a diagram illustrating an example of a hand position information table 1100 storing hand position information generated by the hand detection process according to the first embodiment of the present disclosure. The hand position information stored in the hand position information table 1100 may be generated by a neural network trained to identify the hand position in the hand detection process 600 described with reference to FIG. 6 or the hand detection process 800 described with reference to FIG. 8, for example.

[0100] As shown in FIG. 11, a hand position information table 1100 includes a frame 1101 and hand position information 1102 . In the frame 1101 column, an identifier F 1 ~F N For example, if there are N input images, the first frame is F 1 and the last frame is F M It is. Also, in the column of hand position information 1102, information on bounding boxes that define the position information of the hands detected in each frame is stored. Each bounding box corresponds to a different hand in the frame (i.e., if there are two hands in a frame, there are two bounding boxes for that frame).

[0101] The bounding box that defines the hand position is H m、k where m denotes the frame number (i.e., frame index) and k denotes the bounding box number (i.e., H 1、2 is the first frame F 1 (This is the second bounding box in .)

[0102] Each predicted bounding box is denoted by ((x 1 ",y 1 ''),(x 2 ",y 2 The format is (x '')). 1 ",y 1'') are the coordinates of the upper left corner of the bounding box, and (x 2 ",y 2 '') give the coordinates of the lower right corner of the bounding box.

[0103] As shown in the hand position information table 1100 described above, for each frame constituting an input video, it is possible to manage information on a bounding box that defines the position of a hand detected in that frame.

[0104] Next, a data configuration of a tracking result table that stores information on a tracking result generated by a hand tracking process according to the first embodiment of the present disclosure will be described with reference to FIG.

[0105] As described above, the input image may include the hand of a person other than the hand of the user wearing the photographing device. Therefore, in the present disclosure, a process of tracking the hand is performed to distinguish the hand of the user wearing the photographing device from the hand of another person appearing in the same frame. Fig. 12 is a diagram showing an example of a tracking result table 1200 that stores information on the tracking result generated by the hand tracking process according to the first embodiment of the present disclosure.

[0106] As shown in FIG. 12, a tracking result table 1200 stores a frame 1201 and position information 1202 of the user's hand. In the frame 1201 column, an identifier F 1 ~F N For example, if there are N input images, the first frame is F 1 and the last frame is F M It is. In addition, in the column of user hand position information 1202, information on bounding boxes that define the position information of the user's hands detected in each frame is stored. Each bounding box corresponds to a different hand in the frame (i.e., if two hands are captured in a frame, there are two bounding boxes for that frame). Note that the tracking result table 1200 differs from the hand position information 1102, which includes the position information of all hands in the frame (i.e., hands of people other than the user's hand), in that the tracking result table 1200 includes only the position information of the hand identified by the hand tracking process as the hand of the user wearing the imaging device.

[0107] The bounding box that defines the user's hand position is the WH m、k where m denotes the frame number (i.e., frame index) and k denotes the bounding box number (i.e., WH 1、2 is the first frame F 1 (This is the second bounding box in .)

[0108] Each predicted bounding box is denoted by ((x 1 ",y 1 ''),(x 2 ",y 2 The format is (x '')). 1 ",y 1 '') are the coordinates of the upper left corner of the bounding box, and (x 2 ",y 2 '') give the coordinates of the lower right corner of the bounding box.

[0109] As shown in the tracking result table 1200 described above, for each frame constituting the input video, it is possible to manage information on a bounding box that defines the position of a hand identified as the user's hand in that frame.

[0110] Next, details of the target region determination process according to the first embodiment of the present disclosure will be described with reference to FIG.

[0111] Fig. 13 is a flowchart illustrating an example of a flow of a target area determination process 1300 according to the first embodiment of the present disclosure. The target area determination process 1300 illustrated in Fig. 13 is a process for determining a target area including a user's hand in an input video, and is performed by the above-described determination unit. As described above, the present disclosure is premised on the assumption that the activity to be detected is occurring around the user's hands, and therefore the activity to be detected is likely to be included in the target region that includes the user's hands.

[0112] First, in step S1310, the determination unit calculates the coordinates of the bounding box of the hand (WH 1,1 ,WH 1,2 ,WH 2,1 ,...,WH N,1, WH N,2 The bounding box coordinates of the hand input here are information on the bounding box that defines the position of the user's hand, which is identified by, for example, the above-mentioned hand tracking process, and are stored in the tracking result table 1200 shown in FIG.

[0113] Next, in step S1320, the determination unit declares a variable "n" as a frame index of a frame in the input video, and sets n to 1. This frame index n is a variable that specifies which frame is to be processed among multiple frames included in the input video.

[0114] Next, in step S1330, the determination unit 100 determines whether the frame F n The bounding box of the user's hands in WH n,1 , W.H. n,2 Select the coordinates of.

[0115] Next, in step S1340, the determination unit 100 determines whether the frame F n As mentioned above, we calculate the coordinates of the target region in each bounding box WH m、k is the coordinate of the upper left corner of the bounding box (x t ,y t ) and the coordinates of the bottom right corner of the bounding box (x b ,y b ) as stipulated in Therefore, frame F nIn the example, the bounding box WH of the user's first hand (e.g., the left hand) n,1 is (x n,1,1 ,y n,1,1 ,x n,1,2 ,y n,1,2 ) and the bounding box WH of the user's second hand (e.g., the right hand) n,2 is (x n,2,1 ,y n,2,1 ,x n,2,2 ,y n,2,2 ) coordinates. In this case, the determination unit n Target area in PA n The coordinates of are defined as in Equation 1 below.

number

number

number

number

number

[0116] Next, in step S1350, the decision unit increments the frame index n by 1 (that is, n becomes n+1).

[0117] Next, in step S1360, the determination unit 100 determines whether the frame F nIf the frame index n of the frame to be processed is greater than the number N of frames included in the input video, the process proceeds to step S1370. n If the frame index n is less than or equal to the number N of frames included in the input video, the process returns to step S1330 and repeats the subsequent steps with the next frame.

[0118] Next, in step S1370, after all frames constituting the input video have been processed, the determination unit n For each frame, the coordinates of the bounding box of the target area determined for that frame (PA 1 ,PA 2 ,...,PA N ) to output.

[0119] According to the target area determination process 1300 described above, a target area including an activity to be detected can be determined in each frame constituting the input video.

[0120] Next, a data configuration of a target area information table that stores information on a target area determined by the target area determination process according to the first embodiment of the present disclosure will be described with reference to FIG. FIG. 14 is a diagram illustrating an example of a target area information table 1400 that stores information on a target area determined by the target area determination process according to the first embodiment of the present disclosure.

[0121] As shown in FIG. 14, a target area information table 1400 stores information on a frame 1401 and a target area 1402 . In the frame 1401 column, an identifier F 1 ~F N For example, if there are N input images, the first frame is F 1 and the last frame is F N It is. In addition, in the column of the object area 1402, information on the bounding box of the object area determined in each frame is stored. n where n indicates the frame number (i.e., the frame index).

[0122] As shown in the above-described target region information table 1400, for each frame constituting an input video, the coordinates of the bounding box of the target region including the activity to be detected in that frame can be managed.

[0123] Next, details of the partial image extraction process according to the first embodiment of the present disclosure will be described with reference to FIG.

[0124] Fig. 15 is a flowchart showing a flow of a partial image extraction process 1500 according to the first embodiment of the present disclosure. The partial image extraction process 1500 shown in Fig. 15 is a process for extracting a partial image corresponding to a target region including a user's hand from an input video, and is executed by an extraction unit.

[0125] First, in step S1510, the extraction unit extracts a wide-angle image (F 1 , F 2 , ...F N ) and the coordinates of the bounding box of the hand in each frame (WH 1,1 ,WH 1,2 ,WH 2,1 ,...,WH N,1, WH N,2 ) and the coordinates of the bounding box of the target area determined for each frame (PA 1 ,PA 2 ,...,PA N ) and enter The input image may be, for example, a first-person perspective wide-angle image acquired by the above-mentioned imaging device and received by an input unit of the image processing device.

[0126] The coordinates of the bounding box of the hand in each frame are identified by, for example, the above-mentioned hand tracking process, and are coordinate information stored in the tracking result table 1200 shown in FIG. The coordinates of the bounding box of the target region determined for each frame are determined, for example, by the target region determination process described above, and are coordinate information stored in the target region information table 1400 shown in FIG.

[0127] Next, in step S1520, the extraction unit declares a variable "n" as a frame index of a frame in the input video, and sets n to 1. This frame index n is a variable that specifies which frame to process among multiple frames included in the input video.

[0128] Next, in step S1530, the extraction unit 100 extracts the frame F n The bounding boxes of the user's two hands in n,1 ,WH n,2 The center point c of each 1 , c 2 After calculating the bounding box WH n,1 ,WH n,2 Calculate the distance d between the center points of each of the three (i.e., the distance between the user's hands). As mentioned above, each bounding box WH m、k is the coordinate of the top left corner (x t ,y t ) and the coordinates of the bottom right corner (x b ,y b ) as stipulated in Also, the bounding box WH n,1 Center point c 1 is (x 1 ,y 1 ) and the bounding box WH n,2 Center point c 2 is (x 2 ,y 2 ) is defined by Frame F n Bounding box in WH n,kCenter point c n is calculated by the following formula:

number

number

[0129] Next, in step S1540, the extraction unit compares the distance d calculated in step S1530 with a preset predetermined threshold value T. If the distance d between the user's hands is equal to or smaller than the predetermined threshold value T, the process proceeds to step S1560. On the other hand, if the distance d between the user's hands exceeds the predetermined threshold T, the process proceeds to step S1550.

[0130] In the present disclosure, if the distance d between the user's hands is equal to or less than a predetermined threshold T, it is assumed that both hands of the user are engaged in the same activity, and one partial image including both hands is extracted. Thus, in step S1550, the extraction unit extracts one partial image including both hands from the frame F to be processed. n From the target area PA in the frame n The extraction unit extracts a partial image corresponding to the bounding box of the target region PA. n While maintaining the aspect ratio of the bounding box of the target area PA n Alternatively, a partial image corresponding to a region obtained by multiplying the image by a natural number may be extracted. The size of the partial image extracted here may be enlarged or reduced as appropriate.

[0131] On the other hand, if the distance d between the user's hands exceeds a predefined threshold T, then it is assumed that the hands are engaged in separate activities, and separate partial images are extracted for each hand. n From the bounding box WH of the first user's hand (e.g., the left hand) in the frame n,1 A partial image (e.g., the first partial image) corresponding to the right hand of the second user and a bounding box WH n,1 and a partial image (for example, a second partial image) corresponding to the first partial image are extracted, respectively. The size of the partial image extracted here may be enlarged or reduced as appropriate.

[0132] Next, in step S1570, the extraction unit increments the frame index n by 1 (that is, n becomes n+1).

[0133] Next, in step S1580, the extraction unit 100 extracts the frame F n It is determined whether the frame index n of the frame F to be processed is equal to or smaller than the number N of frames contained in the input video. n If the frame index n of the frame F is greater than the number N of frames included in the input video, the process proceeds to step S1590. n If the frame index n is less than or equal to the number N of frames included in the input video, the process returns to step S1530 and repeats the subsequent steps with the next frame.

[0134] Next, in step S1590, the extraction unit extracts the partial image (I 1,1 , I 1,2 ,...,I N,1 ) to output.

[0135] According to the partial image extraction process 1500 described above, it is possible to determine a target area that is centered on visual information useful for activity detection, such as the area around a user's hand, and to extract a partial image corresponding to the target area from the input video. By performing activity detection on the partial image extracted in this manner, it is possible to perform activity detection with high accuracy.

[0136] Next, with reference to FIG. 16, a data configuration of a partial image management table that stores information on partial images extracted by the partial image extraction process according to the first embodiment of the present disclosure will be described.

[0137] FIG. 16 is a diagram illustrating an example of a partial image management table 1600 that stores information on partial images extracted by the partial image extraction process according to the first embodiment of the present disclosure.

[0138] As shown in FIG. 16, a partial image management table 1600 stores information on a frame 1601 and a partial image 1602 . In the frame 1601 column, an identifier F 1 ~F N For example, if there are N input images, the first frame is F 1 and the last frame is F N It is. In addition, in the column of partial image 1602, information on the partial image extracted from each frame is stored. Each partial image is I n、k where n denotes the frame number (i.e., frame index) and k denotes the subimage number (i.e., I 3、1 is the third frame F 3 (This is the first partial image in .) As described above, if the distance between the hands is equal to or less than a predetermined threshold, one partial image including both hands is extracted from one frame, and if the distance between the hands exceeds the predetermined threshold, a partial image is extracted for each of the hands. Thus, in the partial image management table 1600, each frame is associated with one or two partial images.

[0139] As shown in the partial image management table 1600 described above, for each frame constituting an input video, information on a partial image extracted from that frame can be managed.

[0140] <Example 2> Next, a second embodiment of the present disclosure will be described with reference to FIGS. In the partial image extraction process according to the first embodiment of the present disclosure, when the distance d between the user's hands exceeds a predetermined threshold T, it is assumed that the hands are engaged in different activities, and a configuration has been described in which separate partial images are extracted for each of the hands. However, depending on the task, the user may perform the task using only the dominant hand, and the other hand may not be used for the task. In this case, it is not necessary to extract separate partial images for each of the user's hands.

[0141] Therefore, in the second embodiment of the present disclosure, handedness information that specifies the handedness of the user is acquired, and the handedness of the user in each frame can be identified by identifying the right hand and the left hand of the user in each frame constituting the input video. Then, a partial image is extracted only for the handedness of the user. This makes it possible to detect activity centered on the handedness of the user. In the following, the differences between the second embodiment and the first embodiment of the present disclosure will be mainly described, and a description of the common configuration will be omitted.

[0142] Fig. 17 is a block diagram showing a flow of a target region determination process 1700 according to the second embodiment of the present disclosure. The target region determination process 1700 shown in Fig. 17 is a process for identifying the right and left hands of a user and determining a target region including the right and left hands of the user in an input video, and is performed by the above-mentioned determination unit. As described above, the present disclosure is based on the premise that the activity to be detected is taking place around the user's dominant hand, and therefore the activity to be detected should be included in the target area including the user's dominant hand.

[0143] First, the determination unit inputs an input video 1701, and then executes a process of detecting a hand (one hand or both hands) for a specific frame (e.g., the first frame) in the input video 1701 (1705). Here, the determination unit may use a neural network (e.g., VGG16-SSD or YOLOv3) trained to detect hands based on open source learning data such as Egohands or custom learning data in order to detect the user's hands in the input video 1701. This neural network may be trained to detect not only bare hands, but also hands holding objects such as tools and hands wearing gloves.

[0144] The neural network estimates and outputs a region 1710 containing the hand for each hand in the input video 1705. Here, the neural network may define a bounding box as the region 1710 containing the hand, and output coordinates that define the position of the bounding box.

[0145] Next, the determination unit executes a process of tracking the hand (1715). Here, the determination unit identifies the hand in each frame constituting the input video 1701 based on the region including the hand 1710 estimated for a particular frame. Here, the determination unit executes the process of tracking the hand, thereby being able to determine the region including the user's hand (e.g., a bounding box) for each frame constituting the input video 1701.

[0146] Here, as the process of tracking the hand, the determination unit may use a so-called Kalman filter or a neural network trained to track the hand based on open source learning data or custom learning data. By using this process of tracking the hand, for example, when the hand of another person appears in the input image 1701 in addition to the hand of the user wearing the camera device, it becomes possible to distinguish the hand of the user wearing the camera device from the hand of the other person. In one embodiment, in order to facilitate tracking of the hand, the image processing system may instruct the user to hold their hand straight in front of the camera device when the camera device starts capturing a wide-angle image. This makes it easier to distinguish the hand of the user wearing the camera device from the hand of the other person.

[0147] Next, the determination unit identifies the user's hands in each frame constituting the input video 1701 through a hand tracking process, and then identifies each hand as a left hand or a right hand (1718). Here, to identify the hand as a left hand or a right hand, for example, a neural network or machine learning means trained based on open source or custom learning data may be used. As a result, in each frame constituting the input video 1701, the user's hands are identified as right hands or left hands. The process of identifying a hand as a left hand or a right hand will be described in detail later (see FIG. 18), and therefore will not be described here.

[0148] Next, after identifying the user's hands as right or left hands in each frame constituting the input video 1701, a region of interest 1725 is determined 1720 based on the positions of the regions containing the identified hands (i.e., the coordinates of the bounding boxes). As described above, this region of interest 1725 should contain the activity to be detected.

[0149] According to the target area determination process 1700 described above, both hands of the user can be identified as right hands or left hands in the input video 1705, and a target area including the activity to be detected can be determined.

[0150] 18 is a flowchart showing a flow of a right / left hand discrimination process 1800 according to the second embodiment of the present disclosure. The right / left hand discrimination process 1800 is a process for discriminating both hands of a user as right hands or left hands, and is performed by the above-mentioned determination unit.

[0151] First, in step S1805, the determination unit calculates the coordinates of the bounding box of the hand (WH 1,1 ,WH 1,2 ,WH 2,1 ,...,WH N,1, WH N,2 The bounding box coordinates of the hand input here are information on the bounding box that defines the position of the user's hand, which is identified by, for example, the above-mentioned hand tracking process, and are stored in the tracking result table 1200 shown in FIG.

[0152] Next, in step S1810, the determination unit declares a variable "n" as a frame index of a frame in the input video, and sets n to 1. This frame index n is a variable that specifies which frame to process among multiple frames included in the input video.

[0153] Next, in step S1815, the determination unit 100 determines whether the frame F n If the frame index n of the frame to be processed is 1, the process proceeds to step S1820, and the process determines whether the frame index n of the frame to be processed is 1 (i.e., the first frame of the input video). n If the frame index n is not 1, the process proceeds to step S1835.

[0154] Next, in step S1820, the determination unit 100 determines whether the frame to be processed is F 1 In, the bounding box of the first hand, WH 1,1 is the bounding box of the second hand, WH 1,2 Here, the determination unit determines whether the first hand is located to the left of the bounding box WH1,1 is the bounding box of the second hand, WH 1,2 To determine whether the object is located to the left of the bounding box WH, 1,1 and W.H. 1,2 After calculating the center points of each frame F 1 Calculate the distance to the left edge of

[0155] Then, the determination unit calculates the bounding box WH of the first hand. 1,1 From the center point of frame F 1 The distance to the left edge of the bounding box of the second hand, WH 1,2 From the center point of frame F 1 Determine whether the distance to the left edge of the bounding box of the first hand is shorter than the 1,1 From the center point of frame F 1 The distance to the left edge of the bounding box of the second hand, WH 1,2 From the center point of frame F 1 If the distance to the left edge of the bounding box of the first hand is shorter than the 1,1 is the bounding box of the second hand, WH 1,2 It is determined to be located to the left of

[0156] First hand bounding box WH 1,1 is the bounding box of the second hand, WH 1,2 If the first hand is located to the left of the first hand, the process proceeds to step S1830. 1,1 is the bounding box of the second hand, WH 1,2 If not, the process proceeds to step S1825.

[0157] First hand bounding box WH 1,1 is the bounding box of the second hand, WH 1,2 If the hand is not located to the left of the first hand, in step S1825, the determination unit 1,1 corresponds to the user's right hand, and the bounding box of the second hand, WH1,2 corresponds to the user's left hand. Meanwhile, the bounding box of the first hand, WH 1,1 is the bounding box of the second hand, WH 1,2 If the hand is located to the left of the first hand, in step S1830, the determination unit 1,1 corresponds to the user's left hand, and the bounding box of the second hand, WH 1,2 corresponds to the user's right hand.

[0158] As described above, in the present disclosure, the image processing system instructs the user to hold his / her hands straight in front of the image capture device when the image capture device starts capturing a wide-angle image. Therefore, both hands of the user are captured in frame F 1 In the example shown in Figure 1, the user's left hand is clearly visible in the frame, with the user's left hand located to the left of the right hand. Therefore, it can be inferred that the bounding box of the hand located on the left side of the frame corresponds to the left hand, and the bounding box of the hand located on the right side corresponds to the right hand. After the determinations in steps S1825 and S1830, the process proceeds to step S1850.

[0159] In step S1815, the frame F n If it is determined that the frame index n of the frame F is not 1, the process proceeds to step S1835. n The bounding box of the first hand, WH n,1 and the frame to be processed F n The frame immediately before F n―1 The bounding box LH estimated to correspond to the left hand in n-1 The determination unit calculates the IoU (Intersection over Union) of the frame F n The bounding box of the first hand, WH n,1 and the frame to be processed F n The frame immediately before F n―1The bounding box RH estimated to correspond to the right hand in n-1 Calculate the IoU with.

[0160] IoU is an index that indicates the degree to which two objects overlap. Here, IoU indicates the ratio (amount of overlap) of hand regions in consecutive frames. Frame F to be processed n The bounding box of the first hand, WH n,1 and the frame to be processed F n The frame immediately before F n―1 The bounding box LH estimated to correspond to the left hand in n-1 IoU(IOU(WH n,1、 LH n-1 )) is the frame F to be processed. n The bounding box of the first hand, WH n,1 and the frame to be processed F n The frame immediately before F n―1 The bounding box RH estimated to correspond to the right hand in n-1 IoU(IOU(WH n,1、 RH n-1 If so, processing proceeds to step S1845.

[0161] On the other hand, the frame F n The bounding box of the first hand, WH n,1 and the frame to be processed F n The frame immediately before F n―1 The bounding box LH estimated to correspond to the left hand in n-1 IoU(IOU(WH n,1、 LH n-1 )) is the frame F to be processed. n The bounding box of the first hand, WH n,1 and the frame to be processed F n The frame immediately before F n―1 The bounding box RH estimated to correspond to the right hand in n-1 IoU(IOU(WH n,1、 RHn-1 )) In the following cases, the process proceeds to step S1840.

[0162] As mentioned above, the displacement (i.e., movement) of the user's hand between two consecutive frames is relatively small, so the frame F n The area of ​​both hands in the previous frame F n-1 It overlaps with the area of ​​both hands in the Therefore, IOU(WH n,1、 LH n-1 ) is IOU(WH n,1、 RH n-1 ) or less, in step S1840, the determination unit n,1 corresponds to the user's right hand, and the bounding box of the second hand, WH n,2 corresponds to the user's left hand.

[0163] On the other hand, IOU(WH n,1、 LH n-1 ) is IOU(WH n,1、 RH n-1 If the bounding box WH of the first hand is larger than the bounding box WH of the first hand, in step S1845, n,1 corresponds to the user's left hand, and the bounding box of the second hand, WH n,2 corresponds to the user's right hand. After the determinations of steps S1840 and S1845, the process proceeds to step S1850.

[0164] Next, in step S1850, the determination unit increments the frame index n by 1 (that is, n becomes n+1).

[0165] Next, in step S1855, the determination unit 100 determines whether the frame F n It is determined whether the frame index n of the frame F to be processed is equal to or smaller than the number N of frames contained in the input video. nIf the frame index n of the frame to be processed is greater than the number N of frames included in the input video, the process proceeds to step S1860. On the other hand, if the frame index n of the frame to be processed is equal to or less than the number N of frames included in the input video, the process returns to step S1815 and the subsequent steps are repeated for the next frame.

[0166] Next, in step S1860, the determination unit outputs information about bounding boxes that define the areas of the identified left and right hands of the user for each frame that makes up the input video.

[0167] According to the right / left hand identification process 1800 described above, in each frame constituting the input video 1705, both of the user's hands can be identified as right hands or left hands, and bounding boxes defining the respective areas of the left hand and right hand can be estimated.

[0168] FIG. 19 is a diagram illustrating an example of a right / left hand position information table 1900 storing the position information of the right / left hand of a user generated by the right / left hand identification process according to the second embodiment of the present disclosure.

[0169] As shown in FIG. 19, a right and left hand position information table 1900 stores frames 1901 and position information 1902 of the right and left hands. In the frame 1901 column, an identifier F 1 ~F N For example, if there are N input images, the first frame is F 1 and the last frame is F M It is.

[0170] In addition, the right and left hand position information 1902 column contains bounding boxes (LH n , R.H. n As mentioned above, the bounding boxes that define the positions of the user's right and left hands are ((x 1 ,y 1),(x 2 ,y 2 )), where (x 1 ,y 1 ) are the coordinates of the upper left corner of the bounding box, and (x 2 ,y 2 ) give the coordinates of the lower right corner of the bounding box.

[0171] As shown in the right and left hand position information table 1900 described above, for each frame constituting the input video, it is possible to manage information on the bounding boxes that define the positions of the user's left and right hands identified in that frame.

[0172] Fig. 20 is a flowchart showing an example of the flow of partial image extraction processing 2000 according to the second embodiment of the present disclosure. The partial image extraction processing 2000 shown in Fig. 20 is processing for extracting a partial image corresponding to a target region including a user's hand from an input video based on handedness information indicating the handedness of the user, and is executed by an extraction unit.

[0173] First, in step S2010, the extraction unit extracts a wide-angle image (F 1 , F 2 , ...F N ) and the coordinates of the bounding boxes of the user’s left and right hands in each frame (LH 1 ,RH 1 ,LH 2 ,...,LH N , R.H. N ) and the coordinates of the bounding box of the target area determined for each frame (PA 1 ,PA 2 ,...,PA N ) and handedness information indicating the user's handedness. The input image may be, for example, a first-person perspective wide-angle image acquired by the above-mentioned imaging device and received by an input unit of the image processing device.

[0174] The coordinates of the bounding boxes of the user's right and left hands in each frame are identified, for example, by the above-mentioned right / left hand identification process, and are coordinate information stored in a right / left hand position information table 1900 shown in FIG. The coordinates of the bounding box of the target region determined for each frame are determined, for example, by the target region determination process described above, and are coordinate information stored in the target region information table 1400 shown in FIG. The handedness information is information indicating whether the user is right-handed or left-handed, and may be input by the user via a user interface, which will be described later, for example.

[0175] Next, in step S2020, the extraction unit declares a variable "n" as a frame index of a frame in the input video, and sets n to 1. This frame index n is a variable that specifies which frame is to be processed among multiple frames included in the input video.

[0176] Next, in step S2030, the extraction unit 100 extracts the frame F n The bounding boxes LH of the user's right and left hands n ,RH n The center point c of each L , c R After calculating LH n ,RH n Calculate the distance d between the center points of each of the three (i.e., the distance between the user's hands). Here, the bounding box LH n ,RH n The center point c of each L , c R and the distance d between the center points can be calculated using the above-mentioned formulas 6 and 7, so repeated explanations will be omitted here.

[0177] Next, in step S2040, the extraction unit compares the distance d calculated in step S2030 with a preset predetermined threshold value T. If the distance d between the user's hands is equal to or smaller than the predetermined threshold value T, the process proceeds to step S2050. On the other hand, if the distance d between the user's hands exceeds the predetermined threshold T, the process proceeds to step S2060.

[0178] In the present disclosure, if the distance d between the user's hands is equal to or less than a predetermined threshold T, it is assumed that both hands of the user are engaged in the same activity, and one partial image including both hands is extracted. Thus, in step S2050, the extraction unit extracts one partial image including both hands from the frame F to be processed. n From the bounding box P A of the target area in the frame n The extraction unit extracts a partial image corresponding to the bounding box PA n While maintaining the aspect ratio of the bounding box PA n Alternatively, a partial image corresponding to a region obtained by multiplying the image by a natural number may be extracted. The size of the partial image extracted here may be enlarged or reduced as appropriate.

[0179] On the other hand, if the distance d between the user's hands exceeds a predetermined threshold T, it is assumed that the hands are engaged in different activities. Therefore, in step S2060, the extraction unit extracts partial images only for the user's dominant hand indicated by the dominant hand information input in step S2010. For example, if the user's dominant hand indicated by the dominant hand information is a right hand, the extraction unit derives a bounding box RH n A partial image corresponding to frame F n Extract from.

[0180] On the other hand, if the user's dominant hand indicated by the dominant hand information is the left hand, the extraction unit extracts the bounding box LH of the user's left hand based on the above-mentioned right hand / left hand position information table 1900. n A partial image corresponding to frame F n Extract from. The size of the partial image extracted here may be enlarged or reduced as appropriate.

[0181] Next, in step S2070, the extraction unit increments the frame index n by 1 (that is, n becomes n+1).

[0182] Next, in step S2080, the extraction unit 100 extracts the frame F n It is determined whether the frame index n of the frame F to be processed is equal to or smaller than the number N of frames contained in the input video. n If the frame index n of the frame F is greater than the number N of frames included in the input video, the process proceeds to step S2090. n If the frame index n is less than or equal to the number N of frames contained in the input video, the process returns to step S2030 and repeats the subsequent steps with the next frame.

[0183] Next, in step S2090, the extraction unit extracts the partial image (I 1 , I 1 ,...,I N ) to output.

[0184] According to the partial image extraction process 2000 described above, it is possible to extract partial images based on handedness information indicating the handedness of the user. This makes it possible to omit the process of extracting partial images corresponding to the user's non-handed hand, thereby reducing the processing load and processing time, and enabling activity detection centered on the user's handedness.

[0185] FIG. 21 is a diagram illustrating an example of a partial image management table 2100 that stores information on partial images extracted by the partial image extraction process according to the second embodiment of the present disclosure.

[0186] As shown in FIG. 21, a partial image management table 2100 stores information on a frame 2101 and a partial image 2102 . In the frame 2101 column, an identifier F 1 ~F NFor example, if there are N input images, the first frame is F 1 and the last frame is F N It is. In addition, in the column of partial image 2102, information on the partial image extracted from each frame is stored. Each partial image is I n where n indicates the frame number (i.e., the frame index). As described above, in the second embodiment of the present disclosure, only one partial image (a partial image including both hands of the user or a partial image including the dominant hand of the user) is extracted from each frame, so that in the partial image management table 2100, each frame is associated with one partial image.

[0187] As shown in the partial image management table 2100 described above, for each frame constituting an input video, information on a partial image extracted from that frame can be managed.

[0188] Fig. 22 is a diagram showing a configuration of a user interface 2200 according to the second embodiment of the present disclosure. The user interface 2200 shown in Fig. 22 may be displayed on a screen of the above-mentioned client terminal (for example, the client terminal 210 shown in Fig. 2) and used by a user wearing an image capturing device or a person requesting activity detection.

[0189] As shown in FIG. 22, a user interface 2200 includes an input window 2201, an object frame window 2202, a handedness information window 2203, output windows 2204 and 2205, and user feedback windows 2206 and 2207.

[0190] In the input window 2201, a user of the user interface can select a wide-angle image to be used as an input image. Here, the user of the user interface may select the wide-angle image to be used as an input image from a local storage of a client terminal, a storage unit of an image processing device, a cloud storage, or the like.

[0191] In the target frame window 2202, a user of the user interface may select a frame in the input video to be processed by the image processing device. Here, the user of the user interface may input a frame index of a specific frame, a range of frame indexes that includes multiple frames, or a time range (start time and end time) that includes the frames to be processed.

[0192] In the handedness information window 2203, the user of the user interface can input handedness information indicating the handedness of the user wearing the photographing device. For example, here, the user of the user interface may input whether the handedness of the user wearing the photographing device is right-handed or left-handed.

[0193] In the output window 2204, the user of the user interface can execute the above-mentioned target area determination process for the processing target frame specified in the target frame window 2202 and display the determined target area on the screen. This allows the user of the user interface to check the determined target area on the screen for each frame. In the output window 2205, the user of the user interface can execute the partial image extraction process described above for the target frame specified in the target frame window 2202 and display the extracted partial image on the screen, thereby enabling the user of the user interface to check the extracted partial image on the screen for each frame.

[0194] In the user feedback window 2206, the user of the user interface can specify a destination for saving the extracted partial image and save the partial image. A user of the user interface can process the extracted partial image in the user feedback window 2206. Here, if the extracted partial image does not include the activity to be detected, the user may input an instruction to delete the partial image and extract a new partial image, or manually specify the coordinates of the partial image that includes the activity to be detected.

[0195] According to the image processing means described above, it is possible to perform highly accurate activity detection for a wide-angle image from a first-person perspective. More specifically, in the embodiment of the present disclosure, by using a wide-angle image from a first-person perspective as an input image, it is possible to avoid the conventional problem that the activity to be detected is not included in the shooting range of the wearable camera. In addition, in the embodiment of the present disclosure, by detecting the user's hand and performing activity detection on a partial image of a target area including the detected user's hand, it is possible to perform highly accurate activity detection even when data from other sensors such as GPS cannot be used or when it is desired to detect an activity related to a small object that cannot be identified by GPS data.

[0196] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention. [Explanation of symbols]

[0197] 200 Image Processing System 210 Client terminals 220 Imaging Device 230 users 240 Communication Network 250 Image Processing Device 252 Input section 254 Judgment section 256 Extraction part 258 Detector 260 Output section 262 Storage section

Claims

1. An image processing device, comprising: an input unit that receives a wide-angle image including a user's hand as an input image; a determination unit that determines a target region including the hand in the input image; an extraction unit for extracting a partial image corresponding to the target region from the input image; a detection unit that detects an activity being performed by the user by analyzing the partial image using a predetermined machine learning technique, and generates an activity detection result related to the detected activity; The input image is It consists of several frames, The determination unit is processing a first frame in the input video with a first neural network trained to identify hand positions in images with a standard non-wide angle of view, thereby estimating at least one of a first hand region including a first hand in the first frame and a second hand region including a second hand in the first frame; extracting at least one of a first image corresponding to the first hand region and a second image corresponding to the second hand region from a second frame that is a frame subsequent to the first frame; A predetermined image processing method is applied to at least one of the first image and the second image to generate at least one of a first pseudo-standard image obtained by converting the first image from a wide-angle image to an image with a standard angle of view that is not wide-angle, and a second pseudo-standard image obtained by converting the second image from a wide-angle image to an image with a standard angle of view that is not wide-angle, processing at least one of the first pseudo-standard image and the second pseudo-standard image by the first neural network to estimate at least one of a first hand region including a first hand in the second frame and a second hand region including a second hand in the second frame; 13. An image processing device comprising:

2. An image processing device, comprising: an input unit that receives a wide-angle image including a user's hand as an input image; a determination unit that determines a target region including the hand in the input image; an extraction unit for extracting a partial image corresponding to the target region from the input image; a detection unit that detects an activity being performed by the user by analyzing the partial image using a predetermined machine learning technique, and generates an activity detection result related to the detected activity; The input image is It consists of several frames, The determination unit is generating a pseudo-wide-angle image by converting an image having a standard angle of view, which is not a wide angle, into a wide-angle image by performing a predetermined image processing method on the image having a standard angle of view, training the neural network using learning data in which the pseudo wide-angle image is associated with a teacher image showing a correct hand position in the pseudo wide-angle image, thereby constructing a second neural network trained to identify the hand position in the wide-angle image; processing the input video with the second neural network to estimate at least one of a first hand region including a first hand and a second hand region including a second hand in each frame constituting the input video; 13. An image processing device comprising:

3. The determination unit is determining the target area based on first coordinates defining the first hand area and second coordinates defining the second hand area in the input video in each frame constituting the input video; 3. The image processing device according to claim 1, wherein the image processing device further comprises:

4. The extraction unit is calculating a first center coordinate indicating a center point of the first hand region and a second center coordinate indicating a center point of the second hand region in each frame constituting the input video; Calculating a distance between the first center coordinate and the second center coordinate; If the distance is less than or equal to a predetermined threshold, Extracting a first partial image including the first hand region and the second hand region; If the distance exceeds a predetermined threshold, extracting a first partial image corresponding to the first hand region and a second partial image corresponding to the second hand region; 4. The image processing device according to claim 3, wherein the image processing device further comprises:

5. The input unit includes: When handedness information indicating that the first hand is the handedness of the user is received, The extraction unit is If the distance exceeds a predetermined threshold, extracting a first partial image corresponding to the first hand region; 5. The image processing device according to claim 4, wherein:

6. The determination unit is When the first hand region is located to the left of the second hand region in a first frame of the input video, the first hand region is determined as a region corresponding to a left hand and the second hand region is determined as a region corresponding to a right hand; calculating a first overlap amount indicating a ratio of an overlap between a third hand region in a second frame that is a frame subsequent to the first frame and the first hand region in the first frame; calculating a second overlap amount indicating an overlap rate between a third hand region in the second frame and the second hand region in the first frame; If the first overlap amount is greater than the second overlap amount, the third hand area is determined to be an area corresponding to a left hand; If the second overlap amount is greater than the first overlap amount, the third hand area is determined to be an area corresponding to a right hand.

5. The image processing device according to claim 4, wherein:

7. 1. An image processing method, comprising: receiving a wide-angle image including a user's hand as an input image; A step of generating a pseudo wide-angle image by converting an image having a standard angle of view, which is not a wide-angle image, into a wide-angle image by applying a predetermined image processing method to the image having the standard angle of view; A step of constructing a neural network trained to identify hand positions in a wide-angle image by training the neural network using learning data in which the pseudo wide-angle image is associated with a teacher image that shows a correct hand position in the pseudo wide-angle image; processing the input video by the neural network to estimate at least one of a first hand region including a first hand and a second hand region including a second hand in each frame constituting the input video; determining a target area in each frame constituting the input video based on first coordinates defining the first hand area and second coordinates defining the second hand area in the input video; calculating a first center coordinate indicating a center point of the first hand region and a second center coordinate indicating a center point of the second hand region in each frame constituting the input video; Calculating a distance between the first center coordinate and the second center coordinate; extracting a first partial image including the first hand region and the second hand region if the distance is equal to or less than a predetermined threshold; extracting a first partial image corresponding to the first hand region and a second partial image corresponding to the second hand region if the distance exceeds a predetermined threshold; detecting an activity being performed by the user by analyzing at least one of the first partial image and the second partial image using a predetermined machine learning technique, and generating and outputting an activity detection result related to the detected activity; 13. An image processing method comprising:

8. 1. An image processing system, comprising: An image capturing device for capturing a wide-angle image including a user's hand; an image processing device that detects an activity being performed by the user by analyzing a wide-angle image; a client terminal that confirms a result of the activity detection by the image processing device is connected via a communication network; The image processing device includes: an input unit that receives the wide-angle image as an input image from the imaging device; a determination unit that determines a target area including the user's hand in the input image; an extraction unit for extracting a partial image corresponding to the target region from the input image; a detection unit that detects an activity being performed by the user by analyzing the partial image using a predetermined machine learning technique and generates an activity detection result related to the detected activity; an output unit that outputs the activity detection result to the client terminal; The input image is It consists of several frames, The determination unit is processing a first frame in the input video with a first neural network trained to identify hand positions in images with a standard non-wide angle of view, thereby estimating at least one of a first hand region including a first hand in the first frame and a second hand region including a second hand in the first frame; extracting at least one of a first image corresponding to the first hand region and a second image corresponding to the second hand region from a second frame that is a frame subsequent to the first frame; A predetermined image processing method is applied to at least one of the first image and the second image to generate at least one of a first pseudo-standard image obtained by converting the first image from a wide-angle image to an image with a standard angle of view that is not wide-angle, and a second pseudo-standard image obtained by converting the second image from a wide-angle image to an image with a standard angle of view that is not wide-angle, processing at least one of the first pseudo-standard image and the second pseudo-standard image by the first neural network to estimate at least one of a first hand region including a first hand in the second frame and a second hand region including a second hand in the second frame; 1. An image processing system comprising:

9. An image processing system, comprising: An image capturing device for capturing a wide-angle image including a user's hand; an image processing device that detects an activity being performed by the user by analyzing a wide-angle image; a client terminal that confirms a result of the activity detection by the image processing device is connected via a communication network; The image processing device includes: an input unit that receives the wide-angle image as an input image from the imaging device; a determination unit that determines a target area including the user's hand in the input image; an extraction unit for extracting a partial image corresponding to the target region from the input image; a detection unit that detects an activity being performed by the user by analyzing the partial image using a predetermined machine learning technique and generates an activity detection result related to the detected activity; an output unit that outputs the activity detection result to the client terminal; The input image is It consists of several frames, The determination unit is generating a pseudo-wide-angle image by converting an image having a standard angle of view, which is not a wide angle, into a wide-angle image by performing a predetermined image processing method on the image having a standard angle of view, training the neural network using learning data in which the pseudo wide-angle image is associated with a teacher image showing a correct hand position in the pseudo wide-angle image, thereby constructing a second neural network trained to identify the hand position in the wide-angle image; processing the input video with the second neural network to estimate at least one of a first hand region including a first hand and a second hand region including a second hand in each frame constituting the input video; 1. An image processing system comprising:

10. The determination unit: determining the target area based on first coordinates defining the first hand area and second coordinates defining the second hand area in the input video in each frame constituting the input video; 10. The image processing system according to claim 8 or 9.

11. The extraction unit: calculating a first center coordinate indicating a center point of the first hand region and a second center coordinate indicating a center point of the second hand region in each frame constituting the input video; Calculating a distance between the first center coordinate and the second center coordinate; If the distance is less than or equal to a predetermined threshold, Extracting a first partial image including the first hand region and the second hand region; If the distance exceeds a predetermined threshold, extracting a first partial image corresponding to the first hand region and a second partial image corresponding to the second hand region; 11. The image processing system according to claim 10,

12. The input unit: When handedness information indicating that the first hand is the handedness of the user is received, The extraction unit is If the distance exceeds a predetermined threshold, extracting a first partial image corresponding to the first hand region; 12. The image processing system according to claim 11.

13. The determination unit: When the first hand region is located to the left of the second hand region in a first frame of the input video, the first hand region is determined as a region corresponding to a left hand and the second hand region is determined as a region corresponding to a right hand; calculating a first overlap amount indicating a ratio of an overlap between a third hand region in a second frame that is a frame subsequent to the first frame and the first hand region in the first frame; calculating a second overlap amount indicating an overlap rate between a third hand region in the second frame and the second hand region in the first frame; If the first overlap amount is greater than the second overlap amount, the third hand area is determined to be an area corresponding to a left hand; If the second overlap amount is greater than the first overlap amount, the third hand area is determined to be an area corresponding to a right hand.

12. The image processing system according to claim 11.

Images