Moving image data processing system and moving image data processing method

Abstract

Provided are a moving image data processing system and a moving image data processing method, with which it is possible to identify a cause of an increase in a load applied to an object. This moving image data processing system is provided with an arithmetic circuit (45) capable of accessing moving image data (D1) in which one or more objects that can voluntarily change the posture or motion are captured. The arithmetic circuit (45) detects, from the moving image data (D1), a predetermined posture or motion that applies a load with respect to the one or more objects, classifies the predetermined posture or motion detected from the moving image data (D1) according to the number of occurrences of the predetermined posture or motion and the duration of the predetermined posture or motion, and outputs the result of the classification.

Description

Video data processing system and video data processing method

[0001] The present disclosure relates to a video data processing system and a video data processing method, and more particularly to a video data processing system and a video data processing method for video data.

[0002] Patent Literature 1 discloses a work management system that includes a first calculation unit that calculates a score indicating the degree of workload on a worker from worker information acquired by a first acquisition device, a second calculation unit that calculates a threshold value according to a current situation for at least one of the worker and the work location from at least one of the worker information acquired by the first acquisition device and the work location information acquired by a second acquisition device, and an output unit that outputs information related to the score calculated by the first calculation unit and the threshold value calculated by the second calculation unit.

[0003] Japanese Patent Application Laid-Open No. 2021-162967

[0004] Patent Document 1 states that "when the administrator determines that the workload of a worker is increasing based on information related to the threshold and the worker's score, the administrator can implement measures to reduce the workload of the worker." However, while Patent Document 1 can determine that the workload of a worker is increasing, it does not consider identifying the cause of the increase in workload.

[0005] The present disclosure provides a video data processing system and a video data processing method that enable identification of the cause of an increase in load on an object.

[0006] A video data processing system according to one aspect of the present disclosure includes an arithmetic circuit that can access video data showing one or more objects that can spontaneously change their posture or movement, and the arithmetic circuit detects, from the video data, specific postures or movements that impose stress on the one or more objects, classifies the specific postures or movements detected from the video data according to the number of times the specific postures or movements occur and the duration of the specific postures or movements, and outputs the classification results.

[0007] A video data processing method according to one aspect of the present disclosure is executed by an arithmetic circuit, and detects, from video data showing one or more objects that can spontaneously change their posture or movement, specific postures or movements that are more stressful than a reference posture or movement for one or more objects, classifies the specific postures or movements detected from the video data according to the number of times the specific postures or movements occur and the duration of the specific postures or movements, and outputs the classification results.

[0008] Aspects of the present disclosure allow for the identification of causes of increased loads on an object.

[0009] Schematic diagram of a video data processing system according to an embodiment. Block diagram of a processing device of the video data processing system according to an embodiment. Flowchart of an example of processing by an arithmetic circuit of the video data processing system according to an embodiment. Flowchart of an example of classification processing by the arithmetic circuit of the video data processing system according to an embodiment. Flowchart of a first example of similarity determination of the video data processing system according to an embodiment. Flowchart of a second example of similarity determination of the video data processing system according to an embodiment. Flowchart of a third example of similarity determination of the video data processing system according to an embodiment. Flowchart of a fourth example of similarity determination of the video data processing system according to an embodiment. Flowchart of a fifth example of similarity determination of the video data processing system according to an embodiment. Flowchart of a sixth example of similarity determination of the video data processing system according to an embodiment. Diagram of an example of an analysis screen displayed by the video data processing system according to an embodiment. Diagram of an example of an analysis screen displayed by the video data processing system according to an embodiment.

[0010] [1. Embodiments] Hereinafter, embodiments of the present disclosure will be described, with reference to the drawings where appropriate. However, the following embodiments are merely examples for explaining the present disclosure, and are not intended to limit the present disclosure to the following content (e.g., the shape, dimensions, and arrangement of each component). Positional relationships, such as up, down, left, and right, are based on the positional relationships shown in the drawings unless otherwise specified. Each figure described in the following embodiments is a schematic diagram, and the ratios of the size and thickness of each component in each figure do not necessarily reflect the actual dimensional ratios. Furthermore, the dimensional ratios of each element are not limited to the ratios shown in the drawings.

[0011] In the following description, when it is necessary to distinguish between multiple components, prefixes such as "first" and "second" are added to the names of the components. However, when the components can be distinguished from each other by the symbols attached to them, the prefixes such as "first" and "second" may be omitted in consideration of readability of the text.

[0012] In the following description, when it is necessary to distinguish between multiple components, suffixes such as "-1" and "-2" are added to the symbols of the components. However, when it is not necessary to distinguish between multiple components, the suffixes "-1" and "-2" may be omitted to improve readability.

[0013] 1 is a schematic diagram of a video data processing system 1 according to this embodiment. The video data processing system 1 is used to process video data that shows one or more objects. The video data is obtained in a target space 100, for example.

[0014] The target space 100 is, for example, a facility (e.g., a factory) where products are actually manufactured. Products are manufactured from raw materials through multiple processes. Products include, but are not limited to, various items such as food, pharmaceuticals, electrical appliances, jewelry, furniture, and vehicles. Products are not limited to finished products, but may also be parts. An example of a finished product is an automobile, and an example of a part is a core component of the automobile (such as a piston). Examples of the target space 100 include factories, stores, and buildings (whole buildings, floors). The target space 100 is not limited to manufacturing facilities and may include, for example, non-residential facilities such as offices, buildings, stores, schools, welfare facilities, or medical facilities (such as hospitals), as well as residential facilities such as detached houses, apartment buildings, or individual units of detached houses or apartment buildings. Non-residential facilities also include theaters, movie theaters, public halls, amusement parks, complexes, restaurants, department stores, hotels, inns, kindergartens, libraries, museums, art galleries, underground shopping malls, train stations, and airports. Furthermore, the target space 100 includes not only buildings (structures) but also outdoor facilities such as stadiums, gardens, parking lots, grounds, parks, etc. The target space 100 may also be a mobile facility such as a vehicle, a ship, an airplane, etc.

[0015] An object may spontaneously change its posture or movement. The object may be, for example, a living being such as a person, or an inanimate object such as a robot. The load on the object may change depending on the posture or movement of the object. In this embodiment, the posture is a position indicating the relative positional relationship and spatial relationship of each part of the object. The movement is a change in position indicating the relative positional relationship and spatial relationship of each part of the object. The load on the object may include, for example, a physical load (physical load) on the object. As an example, the object may be a person, particularly a worker working in a facility. When a worker works, various postures or movements occur. The load (physical load) on the worker changes due to the worker's posture or movement. The video data processing system 1 makes it possible to identify the cause of such an increase in the load on the worker.

[0016] The video data processing system 1 includes a camera system 2, a sensor system 3, a processing device 4, and an information terminal 5.

[0017] The information terminal 5 is used by the user 6. The information terminal 5 can be realized, for example, by a personal computer (desktop computer, laptop computer), a mobile terminal (smartphone, tablet terminal, etc.), etc. The video data processing system 1 can present information to the user 6 via the information terminal 5 and, if necessary, request the user 6 to input information.

[0018] The camera system 2 is installed in the target space 100. The camera system 2 is used to acquire video data D1 showing one or more objects that may spontaneously change their posture or movement. The camera system 2 may include multiple cameras 21. The video data D1 is used to identify the cause of an increase in load on the object. The video data D1 may include chronologically ordered image data, time information, and position information.

[0019] As an example, the video data D1 can be used to identify the cause of an increase in the workload of workers working in a factory. In this case, multiple cameras 21 are installed in the factory, and the multiple cameras 21 output video data D1 related to one or more workers. The cameras 21 may or may not be associated one-to-one with the workers. The cameras 21 may capture images of a location where a specific worker works, or may capture images of a location where multiple workers enter and exit. There is no particular limit on the number of cameras 21.

[0020] The sensor system 3 is used to acquire auxiliary data D2 related to one or more objects. The auxiliary data D2, together with the video data D1, is used to identify the cause of an increase in the load on the object. The sensor system 3 may include multiple sensors 31. In this embodiment, the auxiliary data D2 indicates an indicator of the physical load on the object. If the object is a living thing, an example of the indicator of the physical load is biometric data. The biometric data may include data such as heart rate, blood pressure, electromyography, electroencephalogram, and the like. In this case, the sensor 31 may be a biometric sensor such as a heart rate sensor, blood pressure sensor, or electromyography sensor. The sensor 31 may be individually attached to the object. There is no particular limitation on the number of sensors 31.

[0021] The processing device 4 is used to identify the cause of the increase in load on the object. The processing device 4 is connected to the camera system 2, the sensor system 3, and the information terminal 5 so as to be able to communicate with them.

[0022] 2 is a block diagram of the processing device 4. The processing device 4 includes an input device 41, an output device 42, a communication device 43, a storage device 44, and an arithmetic circuit 45. The processing device 4 can be realized by, for example, one or more servers or the like.

[0023] The input device 41 includes one or more human-machine interfaces for inputting information. Examples of the human-machine interface include a keyboard, a pointing device (such as a mouse or a trackball), a touchpad, a position input device for a touch panel display, and the like. The one or more human-machine interfaces of the input device 41 may be built into the processing device 4 or may be externally attached. That is, the input device 41 may include a human-machine interface of the processing device 4 itself and a human-machine interface connected to the processing device 4.

[0024] The output device 42 includes one or more human-machine interfaces for outputting information. Examples of the human-machine interface include a display, a speaker, a touch panel display, and the like. The one or more human-machine interfaces of the output device 42 may be built into the processing device 4 or may be externally attached. That is, the output device 42 may include a human-machine interface of the processing device 4 itself and a human-machine interface connected to the processing device 4.

[0025] The communication device 43 is used for communication through a communication network. The communication device 43 has one or more communication interfaces. The communication device 43 is connectable to a communication network and has the function of communicating through the communication network. The communication device 43 complies with a predetermined communication protocol. The predetermined communication protocol can be selected from various well-known wired and wireless communication standards. In this embodiment, the processing device 4 is communicatively connected to the camera system 2, the sensor system 3, and the information terminal 5 via the communication device 43.

[0026] The storage device 44 includes one or more storages (non-transitory storage media). The storages may be, for example, hard disk drives, optical drives, or solid-state drives (SSDs). The storages may be internal, external, or network-attached storage (NAS).

[0027] The information stored in the storage device 44 includes one or more pieces of video data D1, one or more pieces of auxiliary data D2, and a classification result D3. Fig. 2 shows a state in which the storage device 44 stores the video data D1, the auxiliary data D2, and the classification result D3. The video data D1 and the auxiliary data D2 do not need to be stored in the storage device 44 at all times; they only need to be stored in the storage device 44 when needed by the arithmetic circuit 45.

[0028] The arithmetic circuit 45 is connected to the input device 41, the output device 42, and the communication device 43, and can access the storage device 44. The arithmetic circuit 45 can be realized, for example, by a computer system. The computer system includes one or more processors (microprocessors) and one or more memories. The one or more processors execute programs (stored in one or more memories or the storage device 44) to realize various functions of the processing device 4. The programs may be pre-recorded in the storage device 44, or may be provided via a telecommunications line such as the Internet, or recorded on a non-transitory recording medium such as a memory card.

[0029] The arithmetic circuit 45 executes processing to identify the cause of the increase in the load on the object.

[0030] The process for identifying the cause of an increase in the load on an object will be described below with reference to FIGS.

[0031] FIG. 3 is a flowchart showing an example of processing by the arithmetic circuit 45 of the video data processing system 1 (processing for identifying the cause of an increase in the load on the object).

[0032] The arithmetic circuit 45 acquires video data D1 (S1). When acquiring the video data D1, the arithmetic circuit 45 may acquire the video data D1 from the camera system 2, or may acquire video data D1 that has been previously stored in the storage device 44. The video data D1 may include image data in chronological order and time information.

[0033] The arithmetic circuit 45 detects a predetermined posture or movement for one or more objects from the video data D1 (S2). A predetermined posture or movement is a posture or movement that places a load on the object. Examples of a posture or movement that places a load on the object include a posture or movement that places a load greater than the object's reference posture or movement, or a posture or movement that concentrates a load on a specific part of the object. In this embodiment, the predetermined posture or movement is a crouching posture or crouching movement. For example, when the arithmetic circuit 45 detects a predetermined posture or movement from the video data D1, it may assign an identification number to the detected predetermined posture or movement and store it in the storage device 44 together with the time of occurrence and duration. Table 1 below shows examples of nine predetermined postures or movements detected from the video data D1.

[0034]

[0035] To detect a predetermined posture or movement, the arithmetic circuit 45 determines the skeletal structure of one or more objects shown in the video data D1. A conventionally known skeletal detection technique can be used to determine the skeletal structure. The arithmetic circuit 45 determines whether a predetermined posture or movement is occurring based on the skeletal structure of one or more objects. For example, if the predetermined posture or movement is a crouching posture or movement, the skeletal detection technique can be used to determine the knee flexion angle, and if the flexion angle is 130 degrees or greater, the posture or movement can be determined to be a crouching posture or movement.

[0036] The arithmetic circuit 45 classifies the predetermined postures or movements detected from the video data (S3). The arithmetic circuit 45 classifies the predetermined postures or movements according to the number of occurrences of the predetermined postures or movements and the duration of the predetermined postures or movements. The number of occurrences is an indicator of whether the occurrence of the predetermined postures or movements is steady or sudden. The duration is an indicator of the magnitude of the burden caused by the predetermined postures or movements.

[0037] FIG. 4 is a flowchart showing an example of the classification process performed by the arithmetic circuit 45.

[0038] The arithmetic circuit 45 selects a predetermined posture or movement to be classified from the predetermined postures or movements detected from the video data (S11).

[0039] The arithmetic circuit 45 determines whether the number of occurrences of the predetermined posture or movement to be classified is equal to or greater than a predetermined number (S12). The number of occurrences is determined based on the number of similar predetermined postures or movements among the plurality of predetermined postures or movements detected from the video data D1.

[0040] Next, the determination of similarity of a predetermined posture or movement will be described.

[0041] 5 is a flowchart of a first example of similarity determination by the arithmetic circuit 45. In the first example, whether two or more predetermined postures or movements are similar is determined based on whether the two or more predetermined postures or movements involve the same object. As an example, movements in which workers lift the same object may be determined to be similar, and movements in which workers lift different objects may be determined to be dissimilar.

[0042] First, the arithmetic circuit 45 identifies one or more first images that show a predetermined posture or action to be classified, and then detects one or more objects from the one or more first images (S21).

[0043] The arithmetic circuit 45 selects a target posture or motion to be determined from the target posture or motion detected from the video data, and determines whether the target posture or motion is similar to the target posture or motion, and identifies one or more second images in which the target posture or motion appears. The arithmetic circuit 45 detects one or more objects from the one or more second images (S22).

[0044] The one or more objects are not particularly limited. When the target object is a worker, the one or more objects include an object that is the target of work by the worker, an object used by the worker, etc. Examples of objects that are the target of work by the worker include objects that are the target of assembly or manufacturing by the worker, such as a product, a work in progress, or a part. Examples of objects used by the worker include equipment used by the worker or furniture used by the worker, such as a chair.

[0045] The arithmetic circuit 45 determines whether the same object is present in the one or more first images and the one or more second images (S23). That is, the arithmetic circuit 45 determines whether the one or more objects detected in the one or more second images are the same as the one or more objects detected in the one or more first images. Here, the same object does not necessarily mean an identical object, but may mean an object of the same type.

[0046] If it is determined that the same object exists in one or more first images and one or more second images (S23: YES), the arithmetic circuit 45 determines whether the object is near the target object (S24). That is, if an object is near the target object, it is considered that the object is involved in the target object, but if an object is not near the target object, it is considered that the object is not involved in the target object. Here, whether an object is near the target object can be determined based on criteria such as whether the distance between the center position of the object and the center position of the target object is equal to or less than a predetermined value, or whether the image of the object and the image of the target object overlap.

[0047] If it is determined that the object is near the target object (S24: YES), the arithmetic circuit 45 determines that the predetermined posture or movement of the classification target and the predetermined posture or movement of the judgment target are similar (S25).

[0048] If it is determined that the same object does not exist in one or more first images and one or more second images (S23: NO), or if it is determined that the object is not near the target object (S24: NO), the calculation circuit 45 determines that the specified posture or movement to be classified and the specified posture or movement to be determined are dissimilar (S26).

[0049] In the first example described above, it is determined whether or not the predetermined postures or movements are similar based on the relationship between the object and the target.

[0050] 6 is a flowchart of a second example of similarity determination by the arithmetic circuit 45. In the second example, whether two or more predetermined postures or movements are similar is determined based on whether the two or more predetermined postures or movements occur in the same location. As an example, movements performed by workers in the same location may be determined to be similar, and movements performed by workers in different locations may be determined to be dissimilar.

[0051] First, the arithmetic circuit 45 acquires camera information (S31). The camera information includes information about the position and orientation of the camera 21. For example, if multiple cameras 21 are installed in a factory, the position and orientation of the camera 21 indicate the position and orientation of the camera 21 in the factory. The position and orientation of the camera 21 are used to identify the location in the factory that is captured by the camera 21.

[0052] The arithmetic circuit 45 acquires map information (S32). The map information indicates a map of the location where the camera 21 is installed. For example, if multiple cameras 21 are installed in a factory, the map information includes a map of the factory.

[0053] The arithmetic circuit 45 determines the position of the object taking the predetermined posture or movement to be classified (S33). For example, the arithmetic circuit 45 identifies one or more first images that show the predetermined posture or movement to be classified. The arithmetic circuit 45 determines the position of the object taking the predetermined posture or movement to be classified in the one or more first images. The arithmetic circuit 45 determines the position of the object on the map indicated by the map information from the position of the object taking the predetermined posture or movement to be classified in the one or more first images and the camera information.

[0054] The arithmetic circuit 45 selects a predetermined posture or movement to be determined from the predetermined postures or movements detected from the video data, and determines the position of the object performing the predetermined posture or movement to be determined (S34). For example, the arithmetic circuit 45 identifies one or more second images that show the predetermined posture or movement to be determined. The arithmetic circuit 45 determines the position of the object performing the predetermined posture or movement to be determined in the one or more second images. The arithmetic circuit 45 determines the position of the object performing the predetermined posture or movement to be determined on the map indicated by the map information from the position of the object performing the predetermined posture or movement to be determined in the one or more second images and the camera information.

[0055] The arithmetic circuit 45 determines whether the predetermined posture or movement to be classified and the predetermined posture or movement to be judged occurred in the same location (S35). That is, the arithmetic circuit 45 determines whether the object performing the predetermined posture or movement to be classified and the object performing the predetermined posture or movement to be judged are located in the same location on the map indicated by the map information. The size of the location may be set arbitrarily. The size of the location may be set taking into consideration the degree of positional difference that is acceptable for determining similarity.

[0056] If it is determined that the specified posture or movement to be classified and the specified posture or movement to be judged occurred in the same location (S35: YES), the calculation circuit 45 determines that the specified posture or movement to be classified and the specified posture or movement to be judged are similar (S36).

[0057] If it is determined that the specified posture or movement to be classified and the specified posture or movement to be judged do not occur in the same location (S35: NO), the calculation circuit 45 determines that the specified posture or movement to be classified and the specified posture or movement to be judged are dissimilar (S37).

[0058] In the second example described above, it is determined whether or not a predetermined posture or movement is similar based on the position.

[0059] 7 is a flowchart of a third example of similarity determination by the arithmetic circuit 45. In the third example, whether two or more predetermined postures or movements are similar is determined based on whether the two or more predetermined postures or movements occur in the same time period. As an example, movements performed by workers in the same time period may be determined to be similar, and movements performed by workers in different time periods may be determined to be dissimilar.

[0060] First, the arithmetic circuit 45 determines the first occurrence time of the predetermined posture or movement to be classified (S41). For example, the arithmetic circuit 45 identifies the time corresponding to the image showing the predetermined posture or movement to be classified as the first occurrence time based on the time information of the video data D1.

[0061] The arithmetic circuit 45 selects a target posture or movement to be determined from the detected postures or movements in the video data to determine whether it is similar to the target posture or movement, and determines a second occurrence time of the target posture or movement (S42). For example, the arithmetic circuit 45 identifies, based on the time information in the video data D1, a time corresponding to an image showing the target posture or movement as the second occurrence time.

[0062] The arithmetic circuit 45 determines whether the first occurrence time and the second occurrence time are included in the same time period (S43). The time period may be any length of time, such as one hour or two hours. The time period may be set taking into consideration the amount of time difference that is acceptable for determining similarity.

[0063] If it is determined that the specified posture or movement to be classified and the specified posture or movement to be judged occurred in the same time period (S43: YES), the calculation circuit 45 determines that the specified posture or movement to be classified and the specified posture or movement to be judged are similar (S44).

[0064] If it is determined that the specified posture or movement to be classified and the specified posture or movement to be judged did not occur in the same time period (S43: NO), the calculation circuit 45 determines that the specified posture or movement to be classified and the specified posture or movement to be judged are dissimilar (S45).

[0065] In the third example described above, it is determined whether or not predetermined postures or movements are similar based on time.

[0066] 8 is a flowchart of a fourth example of similarity determination by the arithmetic circuit 45. In the fourth example, whether two or more predetermined postures or movements are similar is determined based on whether the two or more predetermined postures or movements belong to the same object. As an example, movements performed by the same worker may be determined to be similar, and movements performed by different workers may be determined to be dissimilar.

[0067] First, the arithmetic circuit 45 determines a first object that is in a predetermined posture or motion to be classified (S51). For example, the arithmetic circuit 45 identifies the first object that is in a predetermined posture or motion to be classified based on tracking the movement path of the object, image recognition of the object, an identification tag worn by the object, or the like. Conventionally known techniques can be used to identify the object by tracking the movement path of the object, image recognition of the object, or using an identification tag worn by the object, or the like.

[0068] The arithmetic circuit 45 selects a predetermined posture or movement to be judged from the predetermined postures or movements detected from the video data, and determines a second object that performs the predetermined posture or movement to be judged (S52). For example, the arithmetic circuit 45 identifies the second object that performs the predetermined posture or movement to be classified based on tracking of the movement path of the object, an image of the object, an identification tag worn by the object, or the like.

[0069] The arithmetic circuit 45 determines whether the first object and the second object are the same (S53).

[0070] If it is determined that the first object and the second object are the same (S53: YES), the arithmetic circuit 45 determines that the predetermined posture or movement of the classification object and the predetermined posture or movement of the judgment object are similar (S54).

[0071] If it is determined that the first object and the second object are not identical (S53: NO), the calculation circuit 45 determines that the specified posture or movement of the object to be classified and the specified posture or movement of the object to be judged are dissimilar (S55).

[0072] In the fourth example described above, it is determined whether or not a predetermined posture or movement is similar based on the object.

[0073] 9 is a flowchart of a fifth example of similarity determination by the arithmetic circuit 45. In the fifth example, whether two or more predetermined postures or movements are similar is determined based on whether the two or more predetermined postures or movements are the same postures or movements that come before and after each other in a chronological order. As an example, movements that are performed consecutively may be determined to be similar, and movements that are not performed consecutively may be determined to be dissimilar.

[0074] First, the calculation circuit 45 selects the predetermined posture or movement to be judged from the predetermined posture or movement detected from the video data, which is the predetermined posture or movement that comes before or after the predetermined posture or movement to be classified in chronological order (S61).

[0075] The arithmetic circuit 45 determines whether the predetermined posture or movement of the classification target and the predetermined posture or movement of the judgment target are the same (S62). As an example, the arithmetic circuit 45 calculates the degree of agreement between the predetermined posture or movement of the classification target and the predetermined posture or movement of the judgment target. If the degree of agreement is equal to or greater than a predetermined threshold, the arithmetic circuit 45 may determine that the predetermined posture or movement of the classification target and the predetermined posture or movement of the judgment target are the same. Here, the predetermined threshold may be any value, but may be set taking into consideration how much difference is acceptable for determining identity.

[0076] If it is determined that the specified posture or movement to be classified and the specified posture or movement to be judged are the same (S62: YES), the calculation circuit 45 determines that the specified posture or movement to be classified and the specified posture or movement to be judged are similar (S63).

[0077] If it is determined that the specified posture or movement to be classified and the specified posture or movement to be judged are not identical (S62: NO), the calculation circuit 45 determines that the specified posture or movement to be classified and the specified posture or movement to be judged are dissimilar (S64).

[0078] In the fifth example described above, it is determined whether predetermined postures or movements are similar based on whether the same movement or posture occurs consecutively.

[0079] 10 is a flowchart of a sixth example of similarity determination by the arithmetic circuit 45. In the sixth example, whether two or more predetermined postures or movements are similar is determined based on whether auxiliary data related to the two or more predetermined postures or movements is similar. As an example, movements for which auxiliary data is similar may be determined to be similar, and movements for which auxiliary data is not similar may be determined to be dissimilar.

[0080] First, the arithmetic circuit 45 determines first auxiliary data related to the predetermined posture or movement to be classified (S71). The arithmetic circuit 45 identifies the object performing the predetermined posture or movement to be classified and the time when the predetermined posture or movement to be classified occurred. The arithmetic circuit 45 determines, as the first auxiliary data, a portion of the auxiliary data D2 related to the object performing the predetermined posture or movement to be classified, the predetermined time portion including the time when the predetermined posture or movement to be classified occurred. Because the auxiliary data D2 indicates an index of the physical burden on the object, the first auxiliary data indicates an index of the physical burden on the object when performing the predetermined posture or movement to be classified.

[0081] The arithmetic circuit 45 selects a target posture or movement to be determined from the target posture or movement detected from the video data, and determines second auxiliary data related to the target posture or movement (S72). The arithmetic circuit 45 identifies the target object performing the target posture or movement and the time when the target posture or movement occurred. The arithmetic circuit 45 determines, as the second auxiliary data, a portion of the auxiliary data D2 for the target object performing the target posture or movement, the portion of the predetermined time period including the time when the target posture or movement occurred. Because the auxiliary data D2 indicates an index of the physical burden on the target object, the second auxiliary data indicates an index of the physical burden on the target object during the target posture or movement.

[0082] The arithmetic circuit 45 determines whether the first auxiliary data and the second auxiliary data are similar (S73). That is, the arithmetic circuit 45 determines whether the index of physical burden on the object in the predetermined posture or movement to be classified is similar to the index of physical burden on the object in the predetermined posture or movement to be judged. As an example, the arithmetic circuit 45 calculates the degree of agreement between the index of physical burden on the object in the predetermined posture or movement to be classified and the index of physical burden on the object in the predetermined posture or movement to be judged. If the degree of agreement is equal to or greater than a predetermined threshold, the arithmetic circuit 45 may determine that the index of physical burden on the object in the predetermined posture or movement to be classified and the index of physical burden on the object in the predetermined posture or movement to be judged are similar. Here, the predetermined threshold may be any value, as long as it is set taking into account the degree of difference that is acceptable for determining similarity.

[0083] If it is determined that the first auxiliary data and the second auxiliary data are similar (S73: YES), the arithmetic circuit 45 determines that the predetermined posture or movement to be classified and the predetermined posture or movement to be determined are similar (S74).

[0084] If it is determined that the first auxiliary data and the second auxiliary data are not similar (S73: NO), the calculation circuit 45 determines that the specified posture or movement to be classified and the specified posture or movement to be judged are dissimilar (S75).

[0085] In the sixth example described above, it is determined whether or not a predetermined posture or movement is similar based on auxiliary data.

[0086] The calculation circuit 45 may determine that the specified posture or movement is similar if it determines that the posture or movement is similar in any one of the first to sixth examples of the similarity determination described above, or may determine that the specified posture or movement is similar if it determines that the posture or movement is similar in two or more of the first to sixth examples of the similarity determination described above.

[0087] Referring again to FIG. 4 , if it is determined in step S12 that the number of occurrences of the predetermined posture or movement to be classified is equal to or greater than a predetermined number (S12: YES), the arithmetic circuit 45 classifies the predetermined posture or movement to be classified as “frequent” (S13). Here, all predetermined postures or movements similar to the predetermined posture or movement to be classified are classified as “frequent.” In this way, the arithmetic circuit 45 classifies, among the multiple predetermined postures or movements detected from the video data D1, those that occur a predetermined number of times or more into the same type (“frequent”). The predetermined number is not particularly limited, and may be 2. If the predetermined number is 2, step S12 means that “are there any similar predetermined postures or movements?”

[0088] In step S12, if it is determined that the number of occurrences of the specified posture or movement to be classified is less than the predetermined number (S12: NO), the calculation circuit 45 determines whether the duration of the specified posture or movement to be classified is equal to or longer than a predetermined time (S14).

[0089] In step S14, if it is determined that the duration of the predetermined posture or movement to be classified is equal to or longer than the predetermined time (S14: YES), the arithmetic circuit 45 classifies the predetermined posture or movement to be classified as "long duration" (S15). In this way, the arithmetic circuit 45 classifies the predetermined postures or movements detected from the video data D1 that have a duration of equal to or longer than the predetermined time into a different category from the predetermined postures or movements detected from the video data D1 that have a duration of less than the predetermined time. Here, although multiple predetermined postures or movements may be classified as "long duration," this does not mean that the multiple predetermined postures or movements classified as "long duration" are similar to each other.

[0090] In step S14, if it is determined that the duration of the specified posture or movement to be classified is less than the specified time (S14: NO), the arithmetic circuit 45 classifies the specified posture or movement to be classified as "other" (S16).

[0091] After any of steps S13, S15, and S16, the arithmetic circuit 45 determines whether classification of all of the plurality of predetermined postures or movements detected from the video data D1 has been completed (S17). If it determines that classification of all of the plurality of predetermined postures or movements detected from the video data D1 has been completed (S14: YES), the arithmetic circuit 45 ends the classification process. If it determines that classification of all of the plurality of predetermined postures or movements detected from the video data D1 has not been completed (S14: NO), the process returns to step S11, and the arithmetic circuit 45 selects the predetermined postures or movements that have not been classified as targets for classification.

[0092] In this way, the arithmetic circuit 45 generates a classification result D3 of the plurality of predetermined postures or movements detected from the video data D1. The classification result D3 is stored in the storage device 44.

[0093] Table 2 below shows an example of the classification result D3 by the arithmetic circuit 45 for the nine examples of predetermined postures or movements detected from the video data D1 shown in Table 1 above. The classification result D3 includes the type of predetermined posture or movement and the time of occurrence of the predetermined posture or movement. In Table 2 below, the classification result D3 further includes the duration of the predetermined posture or movement. In Table 2 below, the predetermined postures or movements with identification numbers 004 and 005 are classified as long periods of time, but are not similar, and are therefore distinguished as "long period of time 1" and "long period of time 2," respectively.

[0094]

[0095] The arithmetic circuit 45 outputs the classification result D3 (S4). In this embodiment, the arithmetic circuit 45 outputs an analysis screen showing the classification result D3 for a graphical user interface (GUI). The analysis screen includes an object associated with a predetermined posture or movement detected from the video data D1 and a playback window for playing partial video data associated with the object. The partial video data includes a portion of the video data D1 that corresponds to the time at which the predetermined posture or movement associated with the object occurred.

[0096] The analysis screen will be described below. The analysis screen described below corresponds to the classification result D3 shown in Table 2.

[0097] 11 to 14 are explanatory diagrams of examples of analysis screens.

[0098] As shown in FIGS. 11 to 13 , the analysis screen includes an object display window W1. The object display window W1 includes objects O1 to O9 associated with predetermined postures or movements detected from video data. The objects O1 to O9 are associated with predetermined postures or movements with identification numbers 001 to 009, respectively. The objects O1 to O9 are arranged according to their occurrence times. In FIG. 11 , the objects O1 to O9 are arranged in chronological order according to their occurrence times. In particular, the objects O1 to O9 are arranged as points on a scatter plot of occurrence times and durations. In FIG. 11 , the display manner of the objects O1 to O9 varies depending on their type. The display manner of the objects O1 to O9 may include size, shape, color, or changes therein over time. In FIGS. 11 to 13 , objects O1, O8, and O9 corresponding to "frequent" are represented by "□." Objects O2, O3, O5, and O6 corresponding to "other" are represented by "◯." Object O4 corresponding to "long duration 1" is represented by "◇." The object O7 corresponding to "long time 2" is represented by "Δ".

[0099] The analysis screen displays detailed information in response to the selection of an object O1 to O9. The object O1 to O9 can be selected by clicking or rolling over.

[0100] When the object O9 is selected, a message window M1 is displayed in response to the selection of the object O9, as shown in Figure 12. The message window M1 displays detailed information about the object O9.

[0101] The message window M1 includes text T11 related to the object O9. The text T11 includes the type of predetermined posture or movement associated with the object O9 ("Frequent"), the time of occurrence ("16:00"), and a link L11. The link L11 is displayed with the words "Video." When the link L11 is selected, the analysis screen displays a playback window that plays the partial video data associated with the object O9.

[0102] Message window M1 includes additional information regarding another predetermined posture or action of the same type as the predetermined posture or action associated with object O9. The additional information includes text T12 and T13. Text T12 and T13 relate to objects O1 and O8 associated with other predetermined postures or actions (identification numbers 001 and 008) of the same "frequent" type as the predetermined posture or action associated with object O9 (identification number 009).

[0103] The text T12 includes a relationship with the type of predetermined posture or movement associated with the object O9 ("Similar"), the time of occurrence ("8:00"), and a link L12. The link L12 is a link to another partial video data. The other partial video data is a portion of the video data that corresponds to the time of occurrence of another predetermined posture or movement. The link L12 is displayed with the word "video". When the link L12 is selected, the analysis screen displays a playback window that plays the partial video data associated with the object O1.

[0104] The text T13 includes a relationship ("similar") with the type of predetermined posture or movement associated with the object O9, the time of occurrence ("15:00"), and a link L13. The link L13 is a link to another partial video data. The link L13 is displayed with the word "video." When the link L13 is selected, the analysis screen displays a playback window that plays the partial video data associated with the object O8.

[0105] When the object O7 is selected in Fig. 11, a message window M3 is displayed in response to the selection of the object O7, as shown in Fig. 13. The message window M2 displays detailed information about the object O7.

[0106] The message window M2 includes text T2 related to the object O7. The text T2 includes the type of predetermined posture or movement associated with the object O7 ("Long Time 2"), the time of occurrence ("14:00"), and a link L2. The link L2 is displayed with the word "video." When the link L2 is selected, the analysis screen displays a playback window that plays the partial video data associated with the object O7.

[0107] On the analysis screen, the playback window is not displayed at least until an object is selected. This allows for improved utilization efficiency of the analysis screen. As described above, the analysis screen displays detailed information in response to the selection of an object. The detailed information includes a link to partial video data. The playback window is displayed in response to the selection of the link.

[0108] In FIG. 12, when the link L11 is selected, a playback window W2 is displayed as shown in FIG.

[0109] The playback window W2 includes a first display area R1, a second display area R2, and an operation area R3. The first display area R1 is used to display partial video data. The second display area R2 is used to display video data of the skeletal structure of an object obtained from the partial video data. The partial video data and the video data of the skeletal structure are displayed synchronously. The partial video data in the first display area R1 and the skeletal structure in the second display area R2 may be displayed superimposed on each other, thereby integrating the first display area R1 and the second display area R2 into a single window. The operation area R3 is used for operations on the partial video data. In FIG. 14 , the operation area R3 includes buttons B1 to B5 and a seek bar SB1. The buttons B1 to B5 correspond to playback, fast forward, fast rewind, frame forward, and frame rewind, respectively. Operation area R3 is not particularly limited, but may include buttons for selecting a frame of partial video data, changing the FPS of the partial video data, changing the playback magnification of the partial video data, displaying the movement line of an object, etc. Operation area R3 may also include buttons for outputting a heat map, copying the URL of the partial video data, registering an AS (autonomous system), clipping video, etc.

[0110] The video data processing system 1 according to the present embodiment detects stressful postures or movements from video data D1, classifies the detected postures or movements, and outputs the classification results D3. Therefore, measures to reduce the burden on an object can be devised based on the classification of the postures or movements. In particular, the postures or movements are classified according to the number of occurrences and duration of the postures or movements. The number of occurrences serves as an indicator of whether the occurrence of the postures or movements is regular or sudden. For example, regular occurrence of a posture or movement indicates that the object's behavior needs immediate improvement. For example, sudden occurrence of a posture or movement may be considered the result of addressing some abnormality. An increase in the number of sudden occurrences of a posture or movement may indicate an increase in the abnormality. The duration serves as an indicator of the magnitude of the burden caused by the posture or movement. A posture or movement with a long duration requires action, regardless of whether it is regular or sudden. In this way, the video data processing system 1 enables identification of the cause of an increase in the load on an object. Furthermore, the video data processing system 1 is capable of reproducing partial video data for a specific posture or movement from an object showing that posture or movement, making it easier to identify the cause of an increase in the load on an object.

[0111] [1.2 Effects, etc.] The video data processing system 1 described above includes an arithmetic circuit 45 that can access video data D1 that shows one or more objects that can spontaneously change their posture or movement. The arithmetic circuit 45 detects, from the video data D1, specific postures or movements that impose load on the one or more objects, classifies the specific postures or movements detected from the video data D1 according to the number of occurrences of the specific postures or movements and the duration of the specific postures or movements, and outputs the classification results D3. This configuration makes it possible to identify the cause of an increase in load on the object.

[0112] In the video data processing system 1, the classification result D3 includes the type of the predetermined posture or movement and the time when the predetermined posture or movement occurred. This configuration makes it easy to identify the cause of the increase in load on the object.

[0113] In the video data processing system 1, the arithmetic circuit 45 outputs an analysis screen showing the classification result D3. The analysis screen includes objects O1-O9 associated with the predetermined postures or movements detected from the video data D1, and a playback window W2 for playing partial video data associated with the objects O1-O9. The partial video data includes portions of the video data D1 that correspond to the times at which the predetermined postures or movements associated with the objects O1-O9 occurred. This configuration makes it easy to identify the cause of an increase in load on an object.

[0114] In the video data processing system 1, the playback window W2 is not displayed until at least one of the objects O1 to O9 is selected. This configuration makes it possible to improve the efficiency of use of the analysis screen.

[0115] In the video data processing system 1, the objects O1 to O9 are arranged according to the time of occurrence. This configuration makes it easy to identify the cause of an increase in the load on the target object.

[0116] In the video data processing system 1, the objects O1 to O9 are arranged as points on a scatter plot of occurrence times and durations. This configuration makes it easy to identify the cause of an increase in the load on an object.

[0117] In the video data processing system 1, the display modes of the objects O1 to O9 differ depending on the type of predetermined posture or movement. This configuration makes it easy to identify the cause of an increase in the load on the target object.

[0118] In the video data processing system 1, the arithmetic circuit 45 displays detailed information on the analysis screen in response to the selection of one of the objects O1 to O9. The detailed information includes the type of predetermined posture or movement associated with the object O1 to O9. This configuration improves the efficiency of use of the analysis screen.

[0119] In the video data processing system 1, the detailed information includes a link L11 to partial video data. In response to selection of the link L11 included in the detailed information, the arithmetic circuit 45 displays a playback window W2 on the analysis screen. This configuration makes it easy to identify the cause of an increase in the load on the object.

[0120] In the video data processing system 1, the detailed information includes links L12, L13 to other partial video data. In response to selection of a link L12, L13 included in the detailed information, the arithmetic circuit 45 displays another playback window on the analysis screen, which plays back the other partial video data. The other partial video data is a portion of the video data that corresponds to the time of occurrence of another predetermined posture or movement of the same type as the predetermined posture or movement associated with objects O1 to O9. This configuration makes it easy to identify the cause of an increase in load on the target object.

[0121] In the video data processing system 1, among the plurality of predetermined postures or movements detected from the video data D1, the predetermined postures or movements that occur a predetermined number of times or more are classified into the same type. This configuration makes it easy to identify the cause of an increase in the load on the target object.

[0122] In the video data processing system 1, the occurrence count is determined based on the number of similar predetermined postures or movements among a plurality of predetermined postures or movements detected from the video data D1. This configuration makes it easy to identify the cause of an increase in the load on the target object.

[0123] In the video data processing system 1, two or more of a plurality of predetermined postures or movements detected from the video data D1 are considered similar if they involve the same object. This configuration makes it easy to identify the cause of an increase in the load on the object.

[0124] In the video data processing system 1, two or more of the predetermined postures or movements detected from the video data D1 are considered to be similar if they occur in the same location. This configuration makes it easy to identify the cause of an increase in the load on the object.

[0125] In the video data processing system 1, two or more of the predetermined postures or movements detected from the video data D1 are considered to be similar if they occur in the same time period. This configuration makes it easy to identify the cause of an increase in the load on the object.

[0126] In the video data processing system 1, two or more of a plurality of predetermined postures or movements detected from the video data D1 are deemed similar if they belong to the same object. This configuration makes it easy to identify the cause of an increase in the load on the object.

[0127] In the video data processing system 1, two or more of the plurality of predetermined postures or movements detected from the video data D1 are considered similar if they are the same predetermined postures or movements that occur one after the other in a chronological order. This configuration makes it easy to identify the cause of an increase in the load on the target object.

[0128] In the video data processing system 1, one or more objects are humans, and two or more of a plurality of predetermined postures or movements detected from the video data D1 are determined to be similar or not based on an index of the physical burden of the one or more objects. This configuration makes it easy to identify the cause of an increase in the load on the object.

[0129] In the video data processing system 1, among the plurality of predetermined postures or movements detected from the video data D1, those that last for a predetermined time or more are classified into a different type from those that last for less than the predetermined time among the plurality of predetermined postures or movements detected from the video data D1. This configuration makes it easy to identify the cause of an increase in load on the target object.

[0130] In the video data processing system 1, the one or more objects are people. The arithmetic circuit 45 determines the skeletal structure of the one or more objects shown in the video data D1, and determines whether a predetermined posture or movement is occurring based on the skeletal structure of the one or more objects. This configuration enables improved accuracy in detecting the occurrence of a predetermined posture or movement.

[0131] The above-described video data processing system 1 executes the following video data processing method using the arithmetic circuit 45. The video data processing method detects, from video data D1 showing one or more objects that can spontaneously change their posture or movement, specific postures or movements that impose a higher load on one or more objects than a reference posture or movement, classifies the specific postures or movements detected from the video data according to the number of times the specific postures or movements occur and the duration of the specific postures or movements, and outputs the classification results D3. This configuration makes it possible to identify the cause of the increased load on the object.

[0132] The above-described video data processing method is realized by executing a program (computer program) by the arithmetic circuitry 45. This computer program is a program for causing the arithmetic circuitry 45 to execute the above-described video data processing method. This configuration makes it possible to identify the cause of an increase in the load on the object.

[0133] [2. Modifications] The embodiments of the present disclosure are not limited to the above-described embodiments. The above-described embodiments can be modified in various ways depending on the design, etc., as long as the object of the present disclosure can be achieved. Modifications of the above-described embodiments are listed below. The modifications described below can be applied in appropriate combinations.

[0134] In one variation, the predetermined posture or movement is not limited to a crouching posture or movement.

[0135] The predetermined posture or movement may be an arm-raising posture or a movement of raising an arm. The arm-raising posture or the movement of raising an arm can be determined by determining the positions of the shoulder and the elbow or wrist from the video data D1 using skeletal detection technology and comparing the positions of the shoulder with the positions of the elbow or wrist. For example, if the vertical coordinate of the elbow or wrist is greater than the vertical coordinate of the shoulder in the shoulder image, it can be determined that the posture is a lifting posture.

[0136] The predetermined posture or movement may be a posture of bending forward at the waist or a movement of bending forward at the waist, or a posture of rotating the waist or a movement of rotating the waist. The posture of bending forward at the waist or a movement of bending forward at the waist can be determined based on the angle of bending at the waist. The posture of rotating the waist or a movement of rotating the waist can be determined based on the angle difference between the pelvis and the shoulders. The angle of bending at the waist or the angle difference between the pelvis and the shoulders can be determined from the video data D1 using skeletal detection technology and / or three-dimensional posture estimation technology. The angle difference between the pelvis and the shoulders is calculated as the angle between the pelvis line (the line connecting the right hip joint and the left hip joint) and the shoulder line (the line connecting the right shoulder and the left shoulder), and if it is 30 degrees or more, it can be determined to be a posture of rotating the waist or a movement of rotating the waist.

[0137] The predetermined posture or movement may be a heavy object handling action (a movement in which an object is holding a heavy object). The heavy object handling action can be determined based on the positional relationship between the heavy object and the object detected from the video data D1. For example, if the position of the heavy object is changing and there is an object near the heavy object, it can be determined that the action is that the object is holding a heavy object. Note that a trained model can be used to detect heavy objects from the video data D1.

[0138] In one variant, if the object is a person, the predetermined posture or movement may be selected from so-called bad postures.

[0139] In one variation, the computation circuitry 45 may have access to a trained model and may use the trained model to detect a predetermined posture or movement. For example, if the object is a human, the trained model may include one or more trained parameters generated by machine learning using a training dataset that includes one or more object images as input and one or more object postures as output, and an inference program incorporating the one or more trained parameters. The computation circuitry 45 may be configured to input images constituting the video data D1 into the trained model and determine whether a predetermined posture or movement is occurring based on one or more object postures output from the trained model. This allows for improved accuracy in detecting the occurrence of a predetermined posture or movement.

[0140] In this disclosure, terms related to machine learning are defined as follows:

[0141] A "trained model" refers to an "inference program" that incorporates "trained parameters."

[0142] "Trained parameters" refer to parameters (coefficients) obtained as a result of learning using a training dataset. Trained parameters are generated by inputting the training dataset into a training program and mechanically adjusting them for a specific purpose. Although trained parameters are adjusted to suit the purpose of learning, they are simply parameters (numerical information, etc.) on their own, and only function as a trained model when incorporated into an inference program. For example, in the case of deep learning, the main trained parameters are parameters used to weight the links between each node.

[0143] An "inference program" is a program that can output a certain result for an input by applying built-in trained parameters. For example, it is a program that specifies a series of calculation procedures for applying trained parameters acquired as a result of training to an image given as input and outputting a result (authentication or judgment) for that image.

[0144] A "learning dataset," also known as a training dataset, refers to secondary processed data that is generated to facilitate analysis using the target learning method by converting and processing raw data through preprocessing such as removing missing values ​​and outliers, adding separate data such as label information (ground truth data), or a combination of these. A learning dataset may also include data that has been "padded" by applying certain transformations to the raw data.

[0145] "Raw data" refers to data that is primarily acquired by users, vendors, other businesses, research institutions, etc., and that has been converted and processed so that it can be loaded into a database.

[0146] A "learning program" is a program that executes an algorithm to find certain rules from a training dataset and generate a model that expresses those rules. Specifically, this refers to a program that specifies the procedures to be executed by a computer in order to realize learning using the adopted learning method.

[0147] In one variation, the calculation circuitry 45 does not necessarily need to display the classification result D3 on an analysis screen. The output of the classification result D3 may include transmitting, printing, or storing the classification result D3.

[0148] In one modification, the arithmetic circuit 45 can process the video data D1 in real time. In this case, the arithmetic circuit 45 outputs that among the multiple predetermined postures or movements detected from the video data D1, there is a predetermined posture or movement that has occurred less than a predetermined number of times. This makes it possible to deal with any unexpected situation that may occur.

[0149] In one variant, the video data processing system 1 does not necessarily have to include the camera system 2. Similarly, the video data processing system 1 does not necessarily have to include the sensor system 3.

[0150] In one modified example, the processing device 4 and the information terminal 5 of the video data processing system 1 may be configured as a single computer system.

[0151] In one variation, the processing device 4 of the video data processing system 1 may be realized by a computer system such as a plurality of servers. It is not necessary for the plurality of functions (components) of the processing device 4 to be concentrated in a single housing, and the components of the processing device 4 may be distributed across a plurality of housings. Furthermore, at least some of the functions of the processing device 4, for example, some of the functions of the arithmetic circuit 45, may be realized by the cloud (cloud computing) or the like.

[0152] [3. Aspects] As is apparent from the above-described embodiments and modifications, the present disclosure includes the following aspects:

[0032] [Aspect 1] A video data processing system comprising an arithmetic circuit capable of accessing video data showing one or more objects that can spontaneously change their posture or movement, wherein the arithmetic circuit: detects, from the video data, predetermined postures or movements that impose load on the one or more objects; classifies the predetermined postures or movements detected from the video data according to the number of occurrences of the predetermined postures or movements and the durations of the predetermined postures or movements; and outputs the results of the classification.

[0153] [Aspect 2] The video data processing system of Aspect 1, wherein the classification result includes: a type of the predetermined posture or movement; and a time when the predetermined posture or movement occurred.

[0154] [Aspect 3] The video data processing system of Aspect 2, wherein the arithmetic circuit outputs an analysis screen showing the results of the classification, and the analysis screen includes: an object associated with the predetermined posture or movement detected from the video data; and a playback window for playing partial video data associated with the object, and the partial video data includes a portion of the video data corresponding to the time of occurrence of the predetermined posture or movement associated with the object.

[0155] [Aspect 4] The video data processing system according to aspect 3, wherein the playback window is not displayed at least until the object is selected.

[0156] [Aspect 5] The video data processing system according to aspect 3 or 4, wherein the objects are arranged according to the occurrence times.

[0157] [Aspect 6] The video data processing system according to any one of Aspects 3 to 5, wherein the objects are arranged as points on a scatter plot relating to the occurrence times and the durations.

[0158] [Aspect 7] The video data processing system according to any one of Aspects 3 to 6, wherein the display mode of the object differs depending on the type of the predetermined posture or movement.

[0159] [Aspect 8] The video data processing system according to any one of Aspects 3 to 7, wherein the arithmetic circuit displays detailed information on the analysis screen in response to selection of the object, and the detailed information includes the type of the predetermined posture or movement associated with the object.

[0160] [Aspect 9] The video data processing system of aspect 8, wherein the detailed information includes a link to the partial video data, and the arithmetic circuit displays the playback window on the analysis screen in response to selection of the link included in the detailed information.

[0161] [Aspect 10] A video data processing system according to aspect 8 or 9, wherein the detailed information includes a link to another partial video data, and the arithmetic circuit displays, in response to selection of the link included in the detailed information, another playback window on the analysis screen that plays back the other partial video data, and the other partial video data is a portion of the video data that corresponds to the time of occurrence of another predetermined posture or movement of the same type as the predetermined posture or movement associated with the object.

[0162] [Aspect 11] The video data processing system according to any one of Aspects 1 to 10, wherein, of the plurality of predetermined postures or movements detected from the video data, predetermined postures or movements that occur a predetermined number of times or more are classified into the same type.

[0163] [Aspect 12] The video data processing system of Aspect 11, wherein the number of occurrences is determined based on the number of similar predetermined postures or movements among the plurality of predetermined postures or movements detected from the video data.

[0164] [Aspect 13] The video data processing system of Aspect 12, wherein two or more of the plurality of predetermined postures or movements detected from the video data are deemed similar if they involve the same object.

[0165] [Aspect 14] The video data processing system according to Aspect 12 or 13, wherein two or more of the plurality of predetermined postures or movements detected from the video data are considered to be similar if they occur in the same location.

[0166] [Aspect 15] The video data processing system according to any one of Aspects 12 to 14, wherein two or more of the plurality of predetermined postures or movements detected from the video data are deemed to be similar if they occur in the same time period.

[0167] [Aspect 16] The video data processing system according to any one of Aspects 12 to 15, wherein two or more of the plurality of predetermined postures or movements detected from the video data are deemed to be similar if they belong to the same object.

[0168] [Aspect 17] The video data processing system according to any one of Aspects 12 to 16, wherein two or more of the plurality of predetermined postures or movements detected from the video data are considered to be similar if they are the same predetermined postures or movements that come one after the other in a chronological order.

[0169] [Aspect 18] The video data processing system according to any one of Aspects 12 to 17, wherein the one or more objects are humans, and whether two or more of the plurality of predetermined postures or movements detected from the video data are similar or not is determined based on an index of physical strain on the one or more objects.

[0170] [Aspect 19] The video data processing system according to any one of Aspects 1 to 18, wherein, of the plurality of predetermined postures or movements detected from the video data, predetermined postures or movements whose duration is equal to or longer than a predetermined time are classified into a different type from predetermined postures or movements whose duration is shorter than the predetermined time among the plurality of predetermined postures or movements detected from the video data.

[0171] [Aspect 20] The video data processing system of any one of Aspects 1 to 19, wherein the one or more objects are people, and the arithmetic circuit is configured to: determine a skeletal structure of the one or more objects shown in the video data; and determine whether the predetermined posture or movement is occurring based on the skeletal structure of the one or more objects.

[0172] [Aspect 21] The video data processing system of any one of Aspects 1 to 20, wherein the object is a person, the arithmetic circuit has access to a trained model, the trained model comprising: one or more trained parameters generated by machine learning using a training dataset that inputs images of the one or more objects and outputs postures of the one or more objects; and an inference program into which the one or more trained parameters are incorporated, and the arithmetic circuit is configured to input images constituting the video data to the trained model, and determine whether the predetermined posture or movement is occurring based on the postures of the one or more objects output from the trained model.

[0173] [Aspect 22] The video data processing system according to any one of Aspects 1 to 21, wherein the arithmetic circuit outputs that the number of occurrences of the predetermined posture or movement is less than a predetermined number of times among the plurality of predetermined postures or movements detected from the video data.

[0174] [Aspect 23] A video data processing method comprising: detecting, from video data showing one or more objects that can spontaneously change their posture or movement, predetermined postures or movements that are more stressful than a reference posture or movement for the one or more objects; classifying the predetermined postures or movements detected from the video data according to the number of occurrences of the predetermined postures or movements and the duration of the predetermined postures or movements; and outputting the results of the classification.

[0175] [Aspect 24] A program for causing an arithmetic circuit to execute the video data processing method of aspect 23.

[0176] Aspects 2 to 22 are optional elements and are not essential. Aspects 2 to 22 can be appropriately combined with Aspect 23.

[0177] The present disclosure is applicable to a video data processing system, a video data processing method, and a program for processing video data, particularly video data processing systems, video data processing methods, and programs for video data that includes an object that can spontaneously change its posture or movement.

[0178] 1 Video data processing system 45 Arithmetic circuit D1 Video data D2 Auxiliary data D3 Classification results O1 to O9 Objects W2 Playback window

Claims

1. A video data processing system comprising an arithmetic circuit that can access video data showing one or more objects that can spontaneously change their posture or movement, said arithmetic circuit: detects from said video data predetermined postures or movements that impose stress on said one or more objects; classifies said predetermined postures or movements detected from said video data according to the number of occurrences of said predetermined postures or movements and the duration of said predetermined postures or movements; and outputs the results of said classification.

2. The video data processing system of claim 1, wherein the classification results include: the type of the predetermined posture or movement; and the time of occurrence of the predetermined posture or movement.

3. The video data processing system of claim 2, wherein the arithmetic circuit outputs an analysis screen showing the results of the classification, and the analysis screen includes: an object associated with the specific posture or movement detected from the video data; and a playback window for playing partial video data associated with the object, and the partial video data includes a portion of the video data corresponding to the time of occurrence of the specific posture or movement associated with the object.

4. The video data processing system according to claim 3, wherein the objects are arranged as points on a scatter plot of the occurrence times and the durations.

5. The video data processing system according to claim 3, wherein the display mode of the object differs depending on the type of the predetermined posture or movement.

6. The video data processing system of claim 3, wherein the arithmetic circuit displays detailed information on the analysis screen in response to selection of the object, and the detailed information includes the type of the predetermined posture or movement associated with the object.

7. The video data processing system of claim 6, wherein the detailed information includes a link to the partial video data, and the arithmetic circuit displays the playback window on the analysis screen in response to selection of the link included in the detailed information.

8. The video data processing system of claim 6, wherein the detailed information includes a link to another partial video data, and the arithmetic circuit displays another playback window on the analysis screen in response to selection of the link included in the detailed information, which plays back the other partial video data, and the other partial video data is a portion of the video data corresponding to the time of occurrence of another specified posture or movement of the same type as the specified posture or movement associated with the object.

9. The video data processing system according to claim 1, wherein among the plurality of predetermined postures or movements detected from the video data, predetermined postures or movements that occur a predetermined number of times or more are classified into the same type.

10. The video data processing system according to claim 9, wherein the number of occurrences is determined based on the number of similar predetermined postures or movements among the plurality of predetermined postures or movements detected from the video data.

11. The video data processing system according to claim 10, wherein two or more of the plurality of predetermined postures or movements detected from the video data are considered to be similar if they involve the same object.

12. The video data processing system according to claim 10, wherein two or more of the plurality of predetermined postures or movements detected from the video data are considered to be similar if they occur in the same location.

13. The video data processing system according to claim 10, wherein two or more of the plurality of predetermined postures or movements detected from the video data are considered to be similar if they occur in the same time period.

14. The video data processing system according to claim 10, wherein two or more of the plurality of predetermined postures or movements detected from the video data are deemed to be similar if they belong to the same object.

15. The video data processing system of claim 10, wherein two or more of the plurality of predetermined postures or movements detected from the video data are considered to be similar if they are the same predetermined postures or movements that come before or after each other in time series.

16. The video data processing system of claim 10, wherein the one or more objects are humans, and two or more of the plurality of predetermined postures or movements detected from the video data are determined to be similar or not based on an index of physical strain on the one or more objects.

17. The video data processing system of claim 1, wherein, among the plurality of predetermined postures or movements detected from the video data, predetermined postures or movements whose duration is equal to or longer than a predetermined time are classified into a different type from predetermined postures or movements whose duration is less than a predetermined time among the plurality of predetermined postures or movements detected from the video data.

18. The video data processing system of claim 1, wherein the one or more objects are people, and the arithmetic circuit is configured to: determine the skeletal structure of the one or more objects shown in the video data; and determine whether the specified posture or movement is occurring based on the skeletal structure of the one or more objects.

19. The video data processing system of claim 1, wherein the object is a person; the arithmetic circuit has access to a trained model; the trained model comprises: one or more trained parameters generated by machine learning using a training dataset that inputs images of the one or more objects and outputs postures of the one or more objects; and an inference program into which the one or more trained parameters are incorporated; and the arithmetic circuit is configured to input images constituting the video data to the trained model and determine whether the specified posture or movement is occurring based on the postures of the one or more objects output from the trained model.

20. The video data processing system of claim 1, wherein the arithmetic circuit outputs that the number of occurrences of a predetermined posture or movement is less than a predetermined number of times among the plurality of predetermined postures or movements detected from the video data.

21. A video data processing method executed by an arithmetic circuit, which detects, from video data showing one or more objects that can spontaneously change their posture or movement, predetermined postures or movements that are more stressful than a reference posture or movement for the one or more objects, classifies the predetermined postures or movements detected from the video data according to the number of occurrences of the predetermined postures or movements and the duration of the predetermined postures or movements, and outputs the results of the classification.

Images