Information processing device

Abstract

To provide an information processing device capable of identifying a worker with a simple configuration.SOLUTION: In an information processing device 10, an image acquisition unit 11 acquires time-series images including a person and an object, a person position recognition unit 12 recognizes skeletal positions of the person at predetermined time points from the time-series images, an object position recognition unit 13 recognizes positions of the object at the predetermined time points from the time-series images, a calculation unit 14 calculates velocities and accelerations of the person at the predetermined time points, moving distances of the person between the predetermined time points, and positional relations between the person and the object at the predetermined time points, based on the recognized information, and a learning unit 15 generates a model for determining whether or not the person is a worker by machine learning based on the calculated information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to an information processing device. [Background technology]

[0002] In recent years, the demand for high-mix, low-volume production has increased due to the diversification of consumer needs. However, high-mix, low-volume production is less efficient than mass production. To reduce product costs, productivity improvements through work process improvements are required. For this purpose, the introduction of a system to identify workers is essential. However, in high-mix, low-volume production, the workplace becomes highly flexible, and people other than workers, such as developers and designers, are often present in the workplace. Therefore, a system that can easily identify workers is desirable.

[0003] Patent Document 1 discloses a technology that can acquire skeletal information of a single worker even when there are multiple workers present. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2021-162889 [Overview of the project] [Problems that the invention aims to solve]

[0005] Patent Document 1 requires that workers be identified in advance using a facial recognition system, barcode, QP code, etc., before acquiring their skeletal information. Therefore, Patent Document 1 had the problem that workers could not be identified with a simple configuration.

[0006] This disclosure aims to provide an information processing device that can identify workers with a simple configuration, in light of such challenges. [Means for solving the problem]

[0007] The information processing device disclosed herein includes: an image acquisition unit that acquires time-series images including a person and an object; a person position recognition unit that recognizes the skeletal position of the person at a predetermined time from the time-series images; an object position recognition unit that recognizes the position of the object at a predetermined time from the time-series images; a calculation unit that calculates the velocity and acceleration of the person at a predetermined time, the distance the person traveled between predetermined time periods, and the positional relationship between the person and the object at predetermined time, based on the information recognized by the person position recognition unit and the object position recognition unit; and a learning unit that generates a model for determining whether the person is a worker or not by machine learning based on the information calculated by the calculation unit. [Effects of the Invention]

[0008] This disclosure makes it possible to provide an information processing device that can identify workers with a simple configuration. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing an example of the configuration of an information processing device according to the first embodiment. [Figure 2] This is a flowchart showing an example of the learning operation of the information processing device according to the first embodiment. [Figure 3] This figure shows an example of the coordinate positions of each joint of a person recognized in the information processing device according to the first embodiment. [Modes for carrying out the invention]

[0010] (First Embodiment) First, the configuration of the information processing device 10 according to the first embodiment will be described using Figure 1. Figure 1 is a block diagram showing an example of the configuration of an information processing device 10 according to the first embodiment. As shown in Figure 1, the information processing device 10 communicates with the imaging device 1. The imaging device 1 is a device that captures time-series images, such as a camera. The imaging device 1 is installed, for example, in a workplace. The workplace refers to a place where people such as workers, other developers, and designers work.

[0011] The information processing device 10 is a device for discriminating workers in a workplace, such as a PC (Personal Computer). The information processing device 10 includes an image acquisition unit 11, a person position recognition unit 12, an object position recognition unit 13, a calculation unit 14, a learning unit 15, and a discrimination unit 16. The image acquisition unit 11 acquires one or more time-series images including a person from the imaging device 1. The person position recognition unit 12 recognizes the skeletal position of a person in the time-series image at a predetermined time using a skeletal recognition technique such as poseNet. The skeletal position of a person in the time-series image is, for example, the coordinate positions of each joint of the person in the time-series image.

[0012] The object position recognition unit 13 recognizes the position of an object in the time-series image at a predetermined time using an object recognition technique such as YOLO or SSD. The position of the object in the time-series image is, for example, the coordinate position of the object in the time-series image. The calculation unit 14 calculates the speed and acceleration of a person at a predetermined time, the moving distance of the person during a predetermined time period, and the positional relationship between the person and the object at a predetermined time based on the skeletal position of the person and the position of the object in the time-series image at the predetermined time. The learning unit 15 generates a model for discriminating whether a person is a worker by machine learning based on the information calculated by the calculation unit 14. Hereinafter, the generated model is referred to as a learned model.

[0013] Furthermore, the calculation unit 14 calculates the positional relationship between each joint of the person (that is, the posture of the person) at a predetermined time based on the skeletal position of the person and the position of the object in the time-series image at the predetermined time. The learning unit 15 generates a model for discriminating whether a person is a worker and discriminating the work content of the worker by machine learning based on the information calculated by the calculation unit 14.

[0014] The discrimination unit 16 discriminates whether the person in the time-series image is an operator or not by inputting information on the speed and acceleration of the person at a predetermined time, the movement distance of the person during a predetermined time period, the positional relationship between the person and the object at the predetermined time, and the positional relationship between each joint of the person at the predetermined time into the learned model. Further, the discrimination unit 16 discriminates the work content of the operator in the time-series image.

[0015] Next, the operation of the information processing apparatus 10 according to the first embodiment will be described. First, the learning operation of the information processing apparatus 10 according to the first embodiment will be described using FIG. 2. FIG. 2 is a flowchart showing an example of the learning operation of the information processing apparatus 10 according to the first embodiment. In an example of FIG. 2, the imaging device 1 is arranged in the workplace. Further, the imaging device 1 is arranged so that a person is within the angle of view. Then, the imaging device 1 captures one or more time-series images or videos including the person.

[0016] First, in step S101, the image acquisition unit 11 of the information processing apparatus 10 acquires one or more time-series images from the imaging device 1. Note that the image acquisition unit 11 may acquire a video from the imaging device 1. In that case, the image acquisition unit 11 converts the acquired video into one or more time-series images and acquires the converted one or more time-series images.

[0017] Next, in step S102, the person position recognition unit 12 uses a skeleton recognition technology such as poseNet to recognize the coordinate positions of each joint of the person in the time-series image at predetermined times (t1, t2, t3, t4,...) as shown in Table 1. The coordinate positions indicate the x coordinate and the y coordinate in a two-dimensional coordinate system. Here, the coordinate positions of each joint of the person are shown in FIG. 3. FIG. 3 is a diagram showing an example of the coordinate positions of each joint of the person recognized in the information processing apparatus 10 according to the first embodiment.

[0018]

Table 1

[0019] Here, "null" in Table 1 indicates a state where recognition is impossible due to reasons such as part of a joint being hidden. In this embodiment, the person position recognition unit 12 recognizes the x and y coordinates of each joint of the person in a two-dimensional coordinate system. However, the person position recognition unit 12 is not limited to this, and may recognize the z coordinate in addition to the x and y coordinates of each joint of the person in a three-dimensional coordinate system.

[0020] Next, in step S103, the object position recognition unit 13 uses object recognition technology such as YOLO or SSD to recognize the coordinate positions of objects in the time-series images at predetermined times (t1, t2, t3, t4, ...). These coordinate positions represent the x and y coordinates in a two-dimensional coordinate system.

[0021] Next, in step S104, the calculation unit 14 calculates the coordinate positions of each joint of the person at the predetermined time (t0, ..., t) from the coordinate positions of each joint of the person at the predetermined time. final ) at each time t n The velocity v and acceleration a of a person at a given time interval (t0~t final The distance L traveled by the person is calculated at (time t1). Here, for example, time t1 is set to the time when the person's movement began, and time t final This is set to the time when the person stops moving. Note that time t1 and time t final This is not limited to those times; any time may be set. Specifically, the calculation unit 14 calculates the predetermined time (t0, t1, t2, ..., t) as shown in Table 2 below. final In this case, the coordinate position of the joint closest to the waist is selected from the coordinate positions of each joint of the person, and the person's coordinate positions ((X0,Y0), (X1,Y1), (X2,Y2), ..., (X final ,Y final ))

[0022] [Table 2]

[0023] Then, the calculation unit 14 calculates the velocity v and acceleration a of the person at a predetermined time by numerical differentiation with respect to the elapsed time from the coordinate position of the person at the predetermined time, as shown in the following equations (1) and (2). Here, the following equations (1) and (2) represent the calculation formulas for the velocity v and acceleration a of the person at time t1, respectively.

[0024]

Number

[0025]

Number

[0026] Also, the calculation unit 14 calculates the moving distance L of the person by numerical integration with respect to the elapsed time from the coordinate position of the person at a predetermined time, as shown in the following equation (3). The following equation (3) represents the calculation formula for the moving distance L of the person during a predetermined time period (t0 to t final ).

[0027]

Number

[0028] Also, in step S105, the calculation unit 14 calculates the relative position between each joint of the person (that is, the posture of the person) from the coordinate positions of each joint of the person at a predetermined time. Also, in step S106, the calculation unit 14 calculates the relative position of the person with respect to the object at a predetermined time ((X0 - x0, Y0 - y0), ···, (X final ) from the coordinate positions of each joint of the person and the coordinate position of the object at a predetermined time, as shown in the following Table 3. In Table 3, the coordinate positions of the person at a predetermined time (t0, ···, t final - x final , Y final - y final )) are (X0, Y0), ···, (X final ) and the coordinate positions of the person at a predetermined time (t0, ···, t final , Y finalIt is shown as follows. Also, at predetermined times (t0, ..., t final The coordinate positions of the object in ) are (x0, y0), ..., (x final ,y final This is shown as shown above. Note that the calculation unit 14 is not limited to a single object, but may also calculate the relative position of a person with respect to multiple objects.

[0029] [Table 3]

[0030] Furthermore, the calculation unit 14 also calculates the relative positions between each joint of the same person shown in step S105 using the same method as in Table 3.

[0031] Here, the speed, acceleration, and distance traveled of a person calculated by the calculation unit 14 can also be used as features to determine whether or not a person is a worker. For example, workers are often moving around. On the other hand, non-workers are often sitting at workbenches and remaining stationary. Also, the relative position of a person to an object calculated by the calculation unit 14 can be used as a feature to determine whether or not a person is a worker. For example, workers are often near vehicles. On the other hand, non-workers such as developers are often near workbenches. Furthermore, the relative positions between each joint of a person (the person's posture) calculated by the calculation unit 14 can be used as a feature to determine the type of work a person is doing. For example, in assembly work, workers are often seen crawling under vehicles and raising their hands above their shoulders.

[0032] In step S107, the learning unit 15 generates a model by machine learning to determine whether a person is a worker and what kind of work the worker is doing, based on the information calculated by the calculation unit 14 in steps S104 to S106. Specifically, the learning unit 15 creates training data by annotating the information calculated by the calculation unit 14 in steps S104 to S106. The information to be annotated is whether the person is a worker or another person such as a developer or designer. The information to be annotated is also information about the person's work, such as assembly work. Then, the learning unit 15 learns the learning model by machine learning using the training data.

[0033] Next, the discrimination operation of the information processing device 10 according to the first embodiment will be described. The imaging device 1 is placed in the workplace where workers and other people actually work. The imaging device 1 captures one or more time-series images, including people. First, the information processing device 10 performs the processes described in steps S101 to S106 from one or more time-series images. By doing so, the information processing device 10 calculates the velocity and acceleration of the person in the time-series images at a predetermined time, as well as the distance the person travels between predetermined time intervals. The information processing device 10 also calculates the relative positions of each joint of the person and the relative position of the person with respect to an object at a predetermined time.

[0034] The discrimination unit 16 of the information processing device 10 inputs the calculated information into the trained model generated in step S107 described above to determine whether or not the person in the time-series image is a worker. Furthermore, if the person is a worker, the discrimination unit 16 determines the type of work being done.

[0035] As described above, the information processing device 10 according to the first embodiment generates a model for determining whether a person is a worker or not based on information such as the speed, acceleration, and distance traveled of a person in a time-series image, as well as the positional relationship between the person and an object and the positional relationship between each joint of the person, using machine learning. The information processing device 10 then uses the generated trained model to determine whether a person in a time-series image is a worker or not from a time-series image actually captured in the workplace. Therefore, the information processing device 10 does not need to pre-identify workers using a facial recognition system, barcode, QP code, etc., before determining whether a person is a worker or not, and can identify workers with a simple configuration.

[0036] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention.

[0037] Each component of the information processing device 10 in the above-described embodiment is composed of hardware, software, or both, and may consist of one piece of hardware or software, or multiple pieces of hardware or software. Specifically, each component of the information processing device 10 in the above-described embodiment is composed of a computer comprising a processor and memory. The processor may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor may include multiple processors. The memory is composed of a combination of volatile memory and non-volatile memory. The memory may include storage located away from the processor. In this case, the processor may access the memory via an I / O interface not shown. The processor executes one or more programs which include a set of instructions for causing the computer to perform the algorithm described with reference to the drawings.

[0038] The program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored on a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically or otherwise propagating signals. [Explanation of symbols]

[0039] 1 Imaging device, 10 Information processing device, 11 Image acquisition unit, 12 Person position recognition unit, 13 Object position recognition unit, 14 Calculation unit, 15 Learning unit, 16 Discrimination unit

Claims

[Claim 1] An image acquisition unit that acquires time-series images including people and objects, A person position recognition unit that recognizes the skeletal position of the person at a predetermined time from the aforementioned time-series images, An object position recognition unit that recognizes the position of the object at a predetermined time from the aforementioned time-series images, A calculation unit calculates, based on the information recognized by the person position recognition unit and the object position recognition unit, the velocity and acceleration of the person at a predetermined time, the distance the person traveled during a predetermined time interval, and the positional relationship between the person and the object at a predetermined time. The system includes a learning unit that generates a model by machine learning to determine whether the person is a worker or not, based on the information calculated by the calculation unit. Information processing device.

Images