Evaluation method, program, and evaluation system

Abstract

The invention provides an evaluation method, a program, and an evaluation system, which can improve the evaluation precision of the moving direction of a moving body. The evaluation method is executed by an arithmetic circuit capable of accessing a storage device that stores a plurality of learned models. The plurality of learned models include: a simple model (low-resolution model (5)) that learns so as to output an evaluation of the moving direction of the moving body with respect to an input of overall information based on the entire moving body in the image; and detailed models (a medium-resolution model (6), a high-resolution model (7)) that are learned so as to output an evaluation of the direction of movement of the moving body with respect to the input of the overall information and part information based on the part of the moving body. When the resolution of the image of the moving body detected from the target image is less than a threshold value, the evaluation method inputs the overall information to the simple model and outputs an evaluation of the moving direction of the moving body. When the resolution is equal to or greater than a threshold value, the evaluation method inputs the overall information and the part information to the detailed model, and outputs an evaluation of the moving direction of the moving body.

Description

Technical Field

[0001] The present disclosure relates to an evaluation method, a program (computer program), and an evaluation system. Background Art

[0002] Non-patent document 1 discloses a technique for predicting a pedestrian's intention to cross a road using a deep neural network model. In non-patent document 1, non-visual feature quantities are extracted from key points of vehicle speed, pedestrian bounding box, and pedestrian posture, visual feature quantities are extracted from local background of an image and overall background of an image, and the pedestrian's intention to cross a road is predicted based on a feature quantity formed by merging the non-visual feature quantity with the visual feature quantity.

[0003] Prior art literature

[0004] Non-patent literature

[0005] Non-patent document 1: Dongfang Yang et al., “Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention”, [online], April 12, 2021, [retrieved on April 11, 2022], Cornell University, Internet <URL: https: / / arxiv.org / abs / 2104.05485> Summary of the invention

[0006] Technical problem to be solved by the invention

[0007] The technology disclosed in Non-Patent Document 1 is expected to accurately predict the pedestrian's crossing intention by using various information such as vehicle speed, pedestrian's bounding box, pedestrian's posture key points, image local background, and image overall background.

[0008] In the evaluation of the moving direction of a moving object such as the prediction of a pedestrian's crossing intention, the accuracy of the various information itself may also affect the evaluation accuracy of the moving direction of the moving object, as disclosed in Non-Patent Document 1. Therefore, simply increasing the information used in the evaluation of the moving direction of the moving object may not necessarily improve the evaluation accuracy of the moving direction of the moving object.

[0009] The present disclosure provides an evaluation method, a program, and an evaluation system capable of improving the evaluation accuracy of the moving direction of a moving object.

[0010] Technical solutions to solve problems

[0011] An evaluation method according to one embodiment of the present disclosure is executed by a computing circuit capable of accessing a storage device storing a plurality of learned models. The plurality of learned models include: a simple model that is learned in a manner that outputs an evaluation of the moving direction of a moving object relative to input of one or more overall information based on the entire moving object in image information; and a detailed model that is learned in a manner that outputs an evaluation of the moving direction of a moving object relative to input of at least one of the one or more overall information and one or more part information based on one or more parts of the moving object. In the evaluation method, when the resolution of an image of a moving object detected from object image information is less than a threshold, the simple model is selected from the plurality of learned models, and the one or more overall information of the moving object detected from the object image information is input into the simple model, so that the simple model outputs an evaluation of the moving direction of the moving object detected from the object image information. In the evaluation method, when the resolution is greater than the threshold, the detailed model is selected from the plurality of learned models, and at least one of the one or more overall information of the moving object detected from the object image information and one or more part information are input into the detailed model, so that the detailed model outputs an evaluation of the moving direction of the moving object detected from the object image information.

[0012] An evaluation method according to one embodiment of the present disclosure is performed by a computing circuit capable of accessing a storage device storing a plurality of learned models. The plurality of learned models include: a simple model that is learned in a manner that outputs an evaluation of the moving direction of a moving object relative to an input of one or more overall information based on the entire moving object in image information; and a detailed model that is learned in a manner that outputs an evaluation of the moving direction of a moving object relative to at least one of the one or more overall information and an input of one or more part information based on one or more parts of the moving object. In the evaluation method, it is confirmed that the resolution of an image of a moving object detected from object image information is less than a threshold value, and in response to confirming that the resolution is less than the threshold value, a simple model is selected from a plurality of learned models, and one or more overall information of the moving object detected from the object image information is input into the simple model, so that the simple model outputs an evaluation of the moving direction of the moving object detected from the object image information.

[0013] An evaluation method according to one embodiment of the present disclosure is performed by a computing circuit that can access a storage device storing a plurality of learned models. The plurality of learned models include: a simple model that is learned in a manner that outputs an evaluation of the moving direction of a moving object relative to an input of one or more overall information based on the entire moving object in image information; and a detailed model that is learned in a manner that outputs an evaluation of the moving direction of a moving object relative to an input of at least one of the one or more overall information and one or more part information based on one or more parts of the moving object. In the evaluation method, it is confirmed that the resolution of the image of the moving object detected from the object image information is above a threshold value, and in response to confirming that the resolution is above the threshold value, a detailed model is selected from the plurality of learned models, and at least one of the one or more overall information and one or more part information of the moving object detected from the object image information is input into the detailed model, so that the detailed model outputs an evaluation of the moving direction of the moving object detected from the object image information.

[0014] An evaluation method according to one embodiment of the present disclosure is executed by a computing circuit capable of accessing a storage device storing a plurality of learned models. The plurality of learned models include: a first model that is learned in a manner that outputs an evaluation of the moving direction of a moving body relative to input of one or more overall information based on the entire moving body and one or more first part information based on one or more first parts of the moving body in image information; and a second model that is learned in a manner that outputs an evaluation of the moving direction of a moving body relative to input of at least one of the one or more overall information, at least one of the one or more first part information, and one or more second part information based on one or more second parts of the moving body. The one or more second parts are smaller than the one or more first parts. When the resolution of an image of a moving body detected from object image information is less than a threshold value, the evaluation method selects a first model from the plurality of learned models, and inputs one or more overall information and one or more first part information of the moving body detected from the object image information into the first model, so that the first model outputs an evaluation of the moving direction of the moving body detected from the object image information. In the evaluation method, when the resolution is above a threshold, a second model is selected from a plurality of learned models, and at least one of one or more overall information of the moving body detected from the object image information, at least one of one or more first part information, and one or more second part information are input into the second model, so that the second model outputs an evaluation of the moving direction of the moving body detected from the object image information.

[0015] An evaluation method according to one embodiment of the present disclosure is executed by a computing circuit capable of accessing a storage device storing a plurality of learned models. The plurality of learned models include: a first model that is learned in a manner that outputs an evaluation of the moving direction of the moving body relative to input of one or more overall information based on the whole moving body and one or more first part information based on one or more first parts of the moving body in image information; and a second model that is learned in a manner that outputs an evaluation of the moving direction of the moving body relative to input of at least one of the one or more overall information, at least one of the one or more first part information, and one or more second part information based on one or more second parts of the moving body. The one or more second parts are smaller than the one or more first parts. The evaluation method confirms that the resolution of the image of the moving body detected from the object image information is less than a threshold value, selects a first model from the plurality of learned models in response to confirming that the resolution is less than the threshold value, inputs one or more overall information and one or more first part information of the moving body detected from the object image information into the first model, and causes the first model to output an evaluation of the moving direction of the moving body detected from the object image information.

[0016] An evaluation method according to one embodiment of the present disclosure is executed by a computing circuit capable of accessing a storage device storing a plurality of learned models. The plurality of learned models include: a first model that is learned in a manner that outputs an evaluation of the moving direction of a moving object relative to input of one or more overall information based on the entire moving object and one or more first part information based on one or more first parts of the moving object in image information; and a second model that is learned in a manner that outputs an evaluation of the moving direction of a moving object relative to input of at least one of the one or more overall information, at least one of the one or more first part information, and one or more second part information based on one or more second parts of the moving object. The one or more second parts are smaller than the one or more first parts. The evaluation method confirms that the resolution of an image of a moving object detected from object image information is above a threshold value, selects a second model from the plurality of learned models in response to confirming that the resolution is above the threshold value, inputs at least one of the one or more overall information of the moving object detected from the object image information, at least one of the one or more first part information, and one or more second part information into the second model, and causes the second model to output an evaluation of the moving direction of the moving object detected from the object image information.

[0017] A program according to one aspect of the present disclosure is a program for causing a calculation circuit to execute any one of the above-described evaluation methods.

[0018] An evaluation system according to one embodiment of the present disclosure includes: a storage device storing a plurality of learned models; and a computing circuit capable of accessing the storage device. The plurality of learned models include: a simple model that is learned in a manner of outputting an evaluation of the moving direction of a moving object in response to input of one or more overall information based on the entire moving object in image information; and a detailed model that is learned in a manner of outputting an evaluation of the moving direction of a moving object in response to input of at least one of the one or more overall information and one or more part information based on one or more parts of the moving object. When the resolution of an image of a moving object detected from object image information is less than a threshold value, the computing circuit selects a simple model from the plurality of learned models, and inputs one or more overall information of the moving object detected from the object image information into the simple model, so that the simple model outputs an evaluation of the moving direction of the moving object detected from the object image information. When the resolution is greater than the threshold value, the computing circuit selects a detailed model from the plurality of learned models, and inputs at least one of the one or more overall information of the moving object detected from the object image information and one or more part information into the detailed model, so that the detailed model outputs an evaluation of the moving direction of the moving object detected from the object image information.

[0019] An evaluation system according to one embodiment of the present disclosure includes: a storage device storing a plurality of learned models; and a computing circuit capable of accessing the storage device. The plurality of learned models include: a first model that is learned in a manner that outputs an evaluation of the moving direction of a moving body relative to input of one or more overall information based on the entire moving body and one or more first part information based on one or more first parts of the moving body in image information; and a second model that is learned in a manner that outputs an evaluation of the moving direction of a moving body relative to input of at least one of the one or more overall information, at least one of the one or more first part information, and one or more second part information based on one or more second parts of the moving body. The one or more second parts are smaller than the one or more first parts. When the resolution of the image of the moving body detected from the object image information is less than a threshold value, the computing circuit selects a first model from the plurality of learned models, and inputs the one or more overall information and one or more first part information of the moving body detected from the object image information into the first model, so that the first model outputs an evaluation of the moving direction of the moving body detected from the object image information. When the resolution is above the threshold, the operation circuit selects a second model from multiple learned models, and inputs at least one of one or more overall information of the moving body detected from the object image information, at least one of one or more first part information, and one or more second part information into the second model, so that the second model outputs an evaluation of the moving direction of the moving body detected from the object image information.

[0020] (Effects of the Invention)

[0021] The aspect of the present disclosure can improve the accuracy of evaluating the moving direction of a moving object. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a block diagram of a configuration example of a mobile device including an evaluation system according to an embodiment.

[0023] Figure 2 Is Figure 1 A flowchart of an example of a process performed by the evaluation system.

[0024] Figure 3 This is a schematic diagram of a first example of an image of a moving object.

[0025] Figure 4 This is a schematic diagram of a second example of an image of a moving object.

[0026] Figure 5 yes Figure 1 An illustration of the positional relationship between a moving device and a moving body.

[0027] Figure 6 This is a schematic diagram of an example of an image of a moving object at a long distance.

[0028] Figure 7 This is a schematic diagram of an example of an image of a moving object at a medium distance.

[0029] Figure 8 This is a schematic diagram of an example of an image of a moving object at a close distance.

[0030] Fig. 9 yes Figure 1 A schematic diagram illustrating an example of the configuration of a low-resolution model included in an evaluation system.

[0031] Fig.10 yes Figure 1 A schematic diagram illustrating an example of the structure of a medium-resolution model included in an evaluation system.

[0032] Fig.11 yes Figure 1 A schematic diagram illustrating an example of the configuration of a high-resolution model included in an evaluation system.

[0033] Fig.12 yes Figure 1 An illustration of an example of the operation of a mobile device.

[0034] Fig.13 yes Figure 1 An illustration of an example of the operation of a mobile device.

[0035] Fig.14 yes Figure 1 An illustration of an example of the operation of a mobile device. DETAILED DESCRIPTION

[0036] [1. Implementation Method]

[0037] [1.1 Structure]

[0038] Figure 1 This is a block diagram of a configuration example of a mobile device 1 according to one embodiment. The mobile device 1 is, for example, a robot capable of autonomous movement. The mobile device 1 includes an evaluation system 2 , an imaging system 3 , and a control system 4 .

[0039] The evaluation system 2 is a system for evaluating the moving direction (movement intention) of the moving object 10. In other words, the evaluation system 2 is a system for predicting the movement of the moving object 10. In this embodiment, the evaluation system 2 is used to obtain image information of the moving object 10 from the imaging system 3, provide the control system 4 with the evaluation result of the moving direction of the moving object 10, and make the control system 4 perform an action corresponding to the evaluation result of the moving direction of the moving object 10.

[0040] The moving body 10 is a visually recognizable object that is an evaluation object of the evaluation system 2 for the moving direction. The moving body 10 is an object that can move autonomously. That is, the state of the moving body 10 may include a moving state and a stopped state. In the present embodiment, the moving body 10 is a person (a pedestrian as an example). In the following, "moving body" is sometimes expressed as "person", but this is not intended to limit the moving body to a person, but is simply to avoid the complexity of the article and make it easy to understand the description. Therefore, the moving body 10 is not limited to a person, but may also be a living thing such as an animal other than a person. The moving body 10 is not limited to a living thing, but may also be an inanimate object. Examples of inanimate objects include moving objects such as vehicles such as two-wheeled vehicles, cars, ships, and airplanes, and drones. The moving body 10 is not limited to the entire object, but may also be a part of an object.

[0041] The photographing system 3 is a system for generating object image information. The object image information may include data of one or more images of the moving object 10. The object image information may be data of a static image of the moving object 10 or data of a dynamic image of the moving object 10. In the present embodiment, "photographing the moving object 10" includes a case where the moving object 10 is not photographed in its entirety but at least a portion is photographed. For example, in the case where the moving object 10 is a person, it can be said that an image of the face of the person is an image of the moving object 10. The photographing system 3 is connected to the evaluation system 2 in a manner capable of communication, and can provide the evaluation system 2 with object image information. The photographing system 3 includes one or more cameras (digital cameras).

[0042] The control system 4 is a system having a function of executing an action corresponding to the evaluation result of the moving direction of the mobile body 10. In one example, the control system 4 can be used to control the movement (action) of the mobile device 1. The control system 4 can determine the movement of the mobile device 1 based on the evaluation result of the moving direction of the mobile body 10. The control system 4 is connected to the evaluation system 2 in a communicable manner and can receive the evaluation result of the moving direction of the mobile body 10 from the evaluation system 2. The control system 4 includes a computer system, and the computer system includes one or more memories and one or more processors.

[0043] like Figure 1 As shown, the evaluation system 2 includes an interface 21 , a storage device 22 , and a calculation circuit 23 .

[0044] The interface 21 is used to input information to the evaluation system 2 and output information from the evaluation system 2. The interface 21 includes an input-output device 211 and a communication device 212. The input-output device 211 has the function of being an input device for inputting information from a user and an output device for outputting information to a user. The input-output device 211 has one or more human-machine interfaces. Examples of human-machine interfaces include input devices such as keyboards, pointer devices (mouse, trackball, etc.), touch pads, output devices such as displays, speakers, and input-output devices such as touch panels. The communication device 212 is connected to an external device or system in a manner that enables communication. In this embodiment, the communication device 212 is used to communicate with the shooting system 3 and the control system 4 via a communication network. The communication device 212 has one or more communication interfaces. The communication device 212 can be connected to a communication network and has the function of communicating via the communication network. The communication device 112 follows a prescribed communication protocol. The prescribed communication protocol can be selected from various well-known wired and wireless communication standards.

[0045] The storage device 22 is used to store information used by the operation circuit 23 and information generated by the operation circuit 23. The storage device 22 includes one or more storage devices (non-temporary storage media). The storage device may be, for example, any one of a hard disk drive, an optical drive, and a solid state drive (SSD). In addition, the storage device may be any one of a built-in type, an external type, and a NAS (network-attached storage) type.

[0046] The information stored in the storage device 22 includes a plurality of learned models. The plurality of learned models include a low-resolution model 5, a medium-resolution model 6, and a high-resolution model 7. In addition, regarding the low-resolution model 5, the medium-resolution model 6, and the high-resolution model 7, the terms "low", "medium", and "high" are used for the purpose of easily distinguishing them. Figure 1, the storage device 22 is shown to store all of the low-resolution model 5, the medium-resolution model 6, and the high-resolution model 7. The low-resolution model 5, the medium-resolution model 6, and the high-resolution model 7 do not need to be stored in the storage device 22 all the time, and can be stored in the storage device 22 as long as the operation circuit 23 needs to store them.

[0047] The low-resolution model 5, the medium-resolution model 6, and the high-resolution model 7 are used to evaluate the moving direction of the moving object 10. The low-resolution model 5, the medium-resolution model 6, and the high-resolution model 7 will be described in detail later.

[0048] The operation circuit 23 is a circuit for controlling the operation of the evaluation system 2. The operation circuit 23 is connected to the interface 21 and can access the storage device 22 (that is, can access the low-resolution model 5, the medium-resolution model 6, and the high-resolution model 7). The operation circuit 23 can be implemented, for example, by a computer system including one or more processors (microprocessors) and one or more memories. The function of the operation circuit 23 is realized by executing a program (stored in one or more memories or the storage device 22) by one or more processors. Here, the program is recorded in the storage device 22 in advance, but it can also be provided through an electrical communication line such as the Internet, or recorded in a non-temporary recording medium such as a memory card.

[0049] The calculation circuit 23 performs the following operations to evaluate the moving direction of the moving object 10: Figure 2 Evaluation method shown. Figure 2 yes Figure 1 A flowchart of an example of processing of an evaluation system.

[0050] The calculation circuit 23 executes a process of acquiring the target image information ( S11 ). In the present embodiment, the calculation circuit 23 acquires the target image information from the imaging system 3 via the interface 21 .

[0051] Figure 3 1 is a schematic diagram showing a first example of an image of the moving object 10 included in the target image information. Figure 4 2 is a schematic diagram showing a second example of an image of the moving object 10 included in the target image information. Figure 3 and Figure 4 In the example, the moving body 10 moves forward. In other words, the moving body 10 is approaching the moving device 1. Figure 3 In the image, the moving object 10 is moving forward. Figure 4 In the image, the moving object 10 is facing downward and not moving forward.

[0052] The operation circuit 23 performs a process of detecting the moving object 10 from the image included in the object image information (S12). The detection of the moving object 10 from the image can be achieved by a conventionally known method. The detection of the moving object 10 from the image can be achieved, for example, by an image processing technique such as edge detection, or by using a learned model. Figure 3 and Figure 4 As shown, in the present embodiment, the operation circuit 23 extracts a bounding box B representing the moving body 10 from the image. The bounding box B is a rectangular box (rectangular boundary line) that surrounds the moving body 10. In more detail, the bounding box B is a rectangular box of a size that just surrounds the moving body 10. Therefore, the width W and height H of the bounding box B vary according to the size of the moving body 10 on the image. The bounding box B can be set using conventionally known techniques. For example, the bounding box B can be set based on whether the moving body 10 is included in a rectangular area of ​​interest in the image (for example, refer to Japanese Patent Gazette No. 4447245), or can be set to include human bones detected using human bone detection technology (for example, a posture estimation model such as OpenPose).

[0053] The calculation circuit 23 determines the resolution of the image of the moving object 10 detected from the image included in the target image information (S13). In this embodiment, the calculation circuit 23 determines whether the resolution of the image of the moving object 10 is low resolution, medium resolution, or high resolution.

[0054] The resolution of the image of the mobile object 10 corresponds to the number of pixels used to display the mobile object 10 in the image. The resolution of the image of the mobile object 10 may vary, for example, depending on the distance from the camera system 3 to the mobile object 10, the performance (resolution) of the camera of the camera system 3, and the size of the mobile object 10 in real space. The longer the distance from the camera system 3 to the mobile object 10, the smaller the size of the mobile object 10 in the image obtained from the camera system 3, so the number of pixels of the image of the mobile object 10 decreases, and the resolution of the image of the mobile object 10 decreases. When the performance (resolution) of the camera of the camera system 3 is low, the resolution of the image obtained from the camera system 3 itself is low, so the number of pixels of the image of the mobile object 10 decreases, and the resolution decreases. When the performance (resolution) of the camera of the camera system 3 is high, the resolution of the image obtained from the camera system 3 itself is high, so the number of pixels of the image of the mobile object 10 increases, and the resolution becomes high. Even if the distance from the camera system 3 to the mobile object 10 is the same, the resolution of the image of the mobile object 10 changes according to the resolution of the camera of the camera system 3. The size of the moving object 10 in real space can be, for example, the physique (height, etc.) of a person or the height of a car. For example, between an adult with a height of 180 cm and a child with a height of 80 cm, the number of pixels of the child is smaller, so the resolution tends to be lower.

[0055] Figure 5 1 is an explanatory diagram of the positional relationship between the mobile device 1 (camera system 3) and the mobile objects 11, 12, 13. In this embodiment, since the mobile device 1 includes the camera system 3, the distance from the camera system 3 to the mobile objects 11, 12, 13 is equal to the distance from the mobile device 1 to the mobile objects 11, 12, 13. Figure 5 In the figure, the moving body 11 is at a long distance, the moving body 12 is at a medium distance, and the moving body 13 is at a short distance. For example, the distance D1 from the moving device 1 to the moving body 11 is 7 m, the distance D2 from the moving device 1 to the moving body 12 is 5 m, and the distance D3 from the moving device 1 to the moving body 13 is 3 m.

[0056] Figure 6 This is a schematic diagram of an example of an image of a moving object 11 located at a long distance. Figure 7 This is a schematic diagram of an example of an image of the moving object 12 at a medium distance. Figure 8 This is a schematic diagram showing an example of an image of the moving object 13 at a close distance.

[0057] exist Figure 6 The bounding box B1 of the moving object 11, Figure 7 The bounding box B2 of the moving object 12, and Figure 8 In the bounding box B3 of the moving body 13, the area of ​​the bounding box B1 is the smallest, and the area of ​​the bounding box B3 is the largest. The areas of the bounding boxes B1, B2, and B3 are determined by the size of the bounding boxes B1, B2, and B3 relative to the image. Figure 6 , Figure 7 , Figure 8 As shown, the area of ​​the bounding boxes B1, B2, B3 is represented by the product of the number of pixels corresponding to the width W1, W2, W3 of the bounding boxes B1, B2, B3 and the number of pixels corresponding to the height H1, H2, H3 of the bounding boxes B1, B2, B3 in the image.

[0058] The number of pixels constituting the bounding box is proportional to the area of ​​the bounding box. A larger area of ​​the bounding box means a higher resolution of the image of the moving object 10. A smaller area of ​​the bounding box means a lower resolution of the image of the moving object 10. Figure 6 , Figure 7 as well as Figure 8 Among them, the resolution of the image of the moving object 11 is the lowest, the resolution of the image of the moving object 13 is the highest, and the resolution of the moving object 12 is between the resolution of the image of the moving object 11 and the resolution of the image of the moving object 13.

[0059] from Figure 6 , Figure 7 as well as Figure 8It can be seen that the closer the moving object 13 is, the larger the area occupied by the moving object 13 in the image. When the area occupied by the moving object 13 in the image becomes larger, the number of pixels constituting the image of the moving object 13 increases, so the resolution of the moving object 13 becomes higher. Figure 6 , Figure 7 as well as Figure 8 It is clear that the farther the moving object 11 is, the smaller the area occupied by the moving object 11 in the image. When the area occupied by the moving object 11 in the image becomes smaller, the number of pixels constituting the image of the moving object 11 decreases, so the resolution of the moving object 11 becomes lower. In other words, the longer the distance from the shooting system 3 to the moving object 10, the lower the resolution of the image of the moving object 10. The closer the distance from the shooting system 3 to the moving object 10, the higher the resolution of the image of the moving object 10.

[0060] In the present embodiment, the operation circuit 23 determines the resolution of the image of the mobile object 10 based on the area of ​​the bounding box of the mobile object 10. As an example, the operation circuit 23 may use the area of ​​the bounding box of the mobile object 10 as the resolution of the image of the mobile object 10. The operation circuit 23 determines the resolution of the image of the mobile object 10 by comparing the area of ​​the bounding box of the mobile object 10 with a threshold. For example, the operation circuit 23 may determine that the resolution of the image of the mobile object 10 is low resolution when the area of ​​the bounding box is less than a first threshold. In the present embodiment, the operation circuit 23 may determine that the resolution of the image of the mobile object 10 is medium resolution when the area of ​​the bounding box is greater than the first threshold but less than the second threshold. The operation circuit 23 may determine that the resolution of the image of the mobile object 10 is high resolution when the area of ​​the bounding box is greater than the second threshold.

[0061] The first threshold is a threshold for distinguishing between low resolution and medium resolution. For example, when the number of pixels of an image is 360×270, if the number of pixels of a bounding box is less than 30×90, the resolution of the image of the moving object 10 can be considered to be low resolution. In this case, the first threshold is 2700. The second threshold is a threshold for distinguishing between medium resolution and high resolution. For example, when the number of pixels of an image is 360×270, if the number of pixels of a bounding box is greater than 60×180, the resolution of the image of the moving object 10 can be considered to be high resolution. In this case, the second threshold is 10800.

[0062] When the target image information includes a plurality of images, the representative values ​​of the bounding boxes of the moving object 10 captured in the plurality of images can be compared with the threshold values ​​(first threshold value, second threshold value). Examples of the representative values ​​include an average value, a minimum value, a maximum value, a median value, and a mode. Thus, the possibility that the determination of whether the resolution of the image of the moving object 10 corresponds to low resolution, medium resolution, or high resolution changes in a short period of time due to changes in the bounding boxes of the plurality of images can be reduced.

[0063] right Figures 6 to 8 Compared with the moving bodies 11 to 13, Figure 6 In the image of the moving object 11, the whole moving object 11 is relatively clear, but the head and other parts of the moving object 11 are relatively unclear. Figure 6 It is difficult to determine the direction of the face of the moving object 11 with high accuracy. Figure 7 In the image of the moving object 12, the whole moving object 12 is relatively clear, and parts such as the head R of the moving object 11 are also relatively clear. However, parts smaller than the head such as the eyes and skeleton points of the moving object 11 are relatively unclear. Figure 7 It is difficult to determine the line of sight of the moving object 11 or the posture of the moving object 11 with high accuracy. Figure 8 In the image of the moving object 13, the whole moving object 13 is relatively clear, and the parts such as the head R of the moving object 13 (the first part) are also relatively clear. Furthermore, the parts smaller than the head such as the eyes and the skeleton points P of the moving object 13 (the second part) are also relatively clear. Figure 8 The moving body 13 can also determine the line of sight of the moving body 11 or the posture of the moving body 11 with high precision.

[0064] The lower the resolution of the image of the moving object 10, the lower the accuracy of the information obtained from a part of the moving object 10 rather than the entire moving object 10. In addition, the use of low-precision information may cause a decrease in the accuracy of the evaluation of the moving direction of the moving object 10.

[0065] From this point of view, the evaluation system 2 includes a plurality of learned models (low-resolution model 5, medium-resolution model 6, and high-resolution model 7) corresponding to the resolution of the image of the moving object 10. The calculation circuit 23 selects a learned model for evaluating the moving direction of the moving object 10 from the plurality of learned models (low-resolution model 5, medium-resolution model 6, and high-resolution model 7) according to the resolution of the image of the moving object 10.

[0066] More specifically, the operation circuit 23 confirms that the resolution of the image of the moving object 10 detected from the target image information is less than the first threshold, and selects the low-resolution model 5 from the plurality of learned models in response to confirming that the resolution is less than the first threshold. The operation circuit 23 confirms that the resolution of the image of the moving object 10 detected from the target image information is greater than the first threshold, and selects the medium-resolution model 6 or the high-resolution model 7 from the plurality of learned models in response to confirming that the resolution is greater than the first threshold. In this case, the operation circuit 23 confirms that the resolution of the image of the moving object 10 detected from the target image information is less than the second threshold, and selects the medium-resolution model 6 from the plurality of learned models (the medium-resolution model 6 and the high-resolution model 7) in response to confirming that the resolution is less than the second threshold. On the other hand, the operation circuit 23 confirms that the resolution of the image of the moving object 10 detected from the target image information is greater than the second threshold, and selects the high-resolution model 7 from the plurality of learned models (the medium-resolution model 6 and the high-resolution model 7) in response to confirming that the resolution is greater than the second threshold.

[0067] like Figure 2 As shown, when the resolution of the image of the moving object 10 is low resolution ( S13 ; low resolution), the calculation circuit 23 selects the low resolution model 5 ( S21 ).

[0068] The low-resolution model 5 is used when the resolution of the image of the moving object 10 is low. The low-resolution model 5 is a simple model that is learned to output an evaluation of the moving direction of the moving object 10 in response to input based on one or more pieces of overall information of the moving object 10 as a whole in the image information.

[0069] The one or more pieces of overall information include at least one of the position of the mobile object 10, the speed of the mobile object 10, and the image of the entire mobile object 10. The position of the mobile object 10 is, for example, the absolute position of the mobile object 10 in the real space, or the position (relative position) of the mobile object 10 in the real space relative to the mobile device 1. The speed of the mobile object 10 is, for example, the absolute speed of the mobile object 10, or the speed (relative speed) of the mobile object 10 relative to the mobile device 1. The image of the entire mobile object 10 is, for example, an image surrounded by a bounding box corresponding to the mobile object 10.

[0070] In the present embodiment, the evaluation of the moving direction of the mobile body 10 includes the probability that the moving direction of the mobile body 10 is the first moving direction and the probability that the moving direction of the mobile body 10 is the second moving direction. The first moving direction is the direction in which the mobile body 10 is toward the object. The second moving direction is the direction in which the mobile body 10 avoids the object. In the present embodiment, the object is, for example, the mobile device 1. That is, it can be said that the evaluation of the moving direction of the mobile body 10 represents the probability of whether the mobile body 10 will avoid the mobile device 1. From this point of view, it can be said that the evaluation of the moving direction of the mobile body 10 is an avoidance intention score that represents the probability that the mobile body 10 will avoid the mobile device 1. Therefore, it can be said that the evaluation system 2 uses the learned model to perform avoidance intention prediction of the pedestrian in front based on the composition resolution of the human area image.

[0071] Fig. 9 : is a schematic diagram illustrating an example of the configuration of the low-resolution model 5. Fig. 9 In , the object image information includes a plurality of images G in a time series order. A bounding box B of the moving object 10 is detected from the plurality of images G. A plurality of overall information I11, I12, and I13 are obtained from each bounding box B. Thus, the overall information I11 in a time series order, the overall information I12 in a time series order, and the overall information I13 in a time series order are obtained. Fig. 9 In FIG. 1 , the overall information I11 is the position of the moving object 10 , the overall information I12 is the speed of the moving object 10 , and the overall information I13 is an image of the entire moving object 10 .

[0072] Fig. 9 The low-resolution model 5 outputs an evaluation O1 of the moving direction of the moving object 10 with respect to the input of the overall information I11 , I12 , and I13 . Fig. 9 The low-resolution model 5 includes a first network mechanism 511 , a first attention mechanism 512 , a second network mechanism 521 , a second attention mechanism 522 , a third network mechanism 53 , a combining unit 54 and an output unit 55 .

[0073] The first network mechanism 511 extracts feature values ​​from the overall information I11 in the time series order. The first network mechanism 511 includes, for example, a recurrent neural network (RNN) architecture such as a long short-term memory (LSTM) or a gated recurrent unit (GRU). The first attention mechanism 512 includes a time series attention mechanism that determines which overall information I11 in the time series order should be paid attention to. Thus, the feature value of the overall information I11 considered to be the most important in the overall information I11 in the time series order is extracted and input into the combining unit 54.

[0074] The second network mechanism 521 extracts a feature value from the overall information I12 in the time series order. The second network mechanism 521 includes, for example, a recurrent neural network (RNN) architecture such as a long short-term memory (LSTM) or a gated recurrent unit (GRU). The second attention mechanism 522 includes a time series attention mechanism that determines which overall information I12 in the time series order should be paid attention to. Thus, the feature value of the overall information I12 considered to be the most important in the overall information I12 in the time series order is extracted and input into the combining unit 54.

[0075] The third network mechanism 53 extracts a feature value from the overall information I13 in the time series order. As described above, the overall information I13 is an image of the entire moving body 10. The third network mechanism 53 includes, for example, a three-dimensional convolutional neural network (3D-convolutional neural network; 3DCNN) or a convolutional neural network (Convolutional neural network; CNN) and a recurrent neural network (RNN) architecture. Thus, the feature value is extracted from the overall information I13 in the time series order and input to the combining unit 54.

[0076] The combining unit 54 inputs the feature quantities of the overall information I11, I12, and I13 into the output unit 55. Fig. 9 In this case, the combining unit 54 inputs the characteristic value of the overall information I11 considered to be the most important among the overall information I11 in the time series order from the first attention mechanism 512, the characteristic value of the overall information I12 considered to be the most important among the overall information I12 in the time series order from the second attention mechanism 522, and the characteristic value of the overall information I13 in the time series order from the third network mechanism 53 into the output unit 55.

[0077] The output unit 55 outputs the evaluation O1 of the moving direction of the moving object 10 based on the feature amount from the combining unit 54. The output unit 55 includes, for example, a full combining layer. The full combining layer includes, for example, a Softmax function.

[0078] like Figure 2As shown, when the resolution of the image of the moving object 10 detected from the target image information is low resolution (S13; low resolution), the calculation circuit 23 selects a simple model (low resolution model 5) from a plurality of learned models (S21), and uses the low resolution model 5 for evaluation (S22). More specifically, the calculation circuit 23 inputs the overall information I11, I12, and I13 of the moving object 10 detected from the target image information into the low resolution model 5, and causes the low resolution model 5 to output the evaluation O1 of the moving direction of the moving object 10 detected from the target image information. Thus, in the case of low resolution, the evaluation system 2 evaluates the moving direction of the moving object 10 without using information (part information described later) with a high probability of low accuracy. As a result, the evaluation system 2 can improve the evaluation accuracy of the moving direction of the moving object 10.

[0079] When the resolution of the image of the moving object 10 is medium resolution (S13; medium resolution), the calculation circuit 23 selects the medium resolution model 6 (S31). Figure 2 As shown, when the resolution of the image of the moving object 10 is high resolution ( S13 ; high resolution), the calculation circuit 23 selects the high resolution model 7 ( S41 ).

[0080] The medium-resolution model 6 and the high-resolution model 7 are respectively used when the resolution of the image of the moving object 10 is medium resolution. The medium-resolution model 6 is a detailed model that is learned in a manner that outputs an evaluation of the moving direction of the moving object 10 with respect to input of at least one of one or more overall information based on the entire moving object 10 in the image information and one or more part information based on one or more parts of the moving object 10 in the image information.

[0081] The one or more part information includes, for example, at least one of the position of one or more parts of the mobile body 10, the orientation of one or more parts of the mobile body 10, the image of one or more parts of the mobile body 10, and the information based on the association of multiple parts of the mobile body 10. As an example of the position of the part, the position of the face of the mobile body 10 can be cited. As an example of the orientation of the part, the orientation of the face of the mobile body and the line of sight of the mobile body can be cited. As an example of the image of the part, the image of the face of the mobile body can be cited. As an example of the information based on the association of multiple parts, information related to the posture of the mobile body can be cited.

[0082] The parts of the moving body 10 can be classified into a first part and a second part smaller than the first part. The first part is a part of the body of the moving body 10, for example Figure 7The first part is not limited to the face, and may be the head, arm, trunk, leg, etc. The second part is a part smaller than the body part of the moving object 10. Compared with the first part, the second part can be regarded as a point in the moving object 10. Examples of the second part include Figure 8 The skeleton point P is shown. The second part is not limited to the skeleton point, and can also be eyes, nose, fingers, joints, etc.

[0083] As can be seen from this, the part information can be classified into first part information based on the first part and second part information based on the second part. In order to obtain the second part information with high accuracy, the resolution of the image of the moving object 10 needs to be higher than that of the first part information.

[0084] In this embodiment, the medium-resolution model 6 uses the first part information in addition to the overall information. That is, the medium-resolution model 6 is a first model that is learned in a manner of outputting an evaluation of the moving direction of the moving object 10 with respect to input of one or more overall information based on the entire moving object 10 and one or more first part information based on one or more first parts of the moving object 10 in the image information.

[0085] In the present embodiment, the high-resolution model 7 uses the second part information in addition to the overall information and the first part information. That is, the high-resolution model 7 is a second model that is learned in such a manner as to output an evaluation of the moving direction of the moving object 10 with respect to input of at least one overall information based on one or more overall information of the moving object 10 as a whole, at least one first part information of one or more first part information, and one or more second part information in the image information.

[0086] Fig.10 6 is a schematic diagram illustrating an example of the configuration of a medium-resolution model 6. Fig.10 In , the object image information includes a plurality of images G in a time series order. A bounding box B of the moving object 10 is detected from the plurality of images G. A plurality of overall information I11, I12, and I13 are obtained from each bounding box B. Thus, the overall information I11 in a time series order, the overall information I12 in a time series order, and the overall information I13 in a time series order are obtained. Fig.10 In FIG. 1 , the overall information I11 is the position of the moving object 10 , the overall information I12 is the speed of the moving object 10 , and the overall information I13 is an image of the entire moving object 10 .

[0087] exist Fig.10In , a bounding box R of the face of the moving object 10 is detected from a plurality of images G. A plurality of first part information I21, I22, and I23 are obtained from each bounding box R. Thus, the first part information I21 in time series order, the first part information I22 in time series order, and the first part information I23 in time series order are obtained. Fig.10 In the figure, the first part information I21 is the position of the face of the moving object 10 , the first part information I22 is the orientation of the face of the moving object 10 , and the first part information I23 is the image of the face of the moving object 10 .

[0088] Fig.10 The medium-resolution model 6 outputs an evaluation O2 of the moving direction of the moving body 10 based on the input of the overall information I11 , I12 , I13 and the part information I21 , I22 , I23 . Fig.10 The medium-resolution model 6 includes a first network mechanism 611, a first attention mechanism 612, a second network mechanism 621, a second attention mechanism 622, a third network mechanism 63, a fourth network mechanism 641, a fourth attention mechanism 642, a fifth network mechanism 651, a fifth attention mechanism 652, a sixth network mechanism 66, a combination unit 67 and an output unit 68.

[0089] The first network mechanism 611, the first attention mechanism 612, the second network mechanism 621, the second attention mechanism 622 and the third network mechanism 63 are respectively Fig. 9 The first network mechanism 511, the first attention mechanism 512, the second network mechanism 521, the second attention mechanism 522 and the third network mechanism 53 of the low-resolution model 5 are the same.

[0090] The fourth network mechanism 641 extracts a feature value from the first part information I21 in the time series order. The fourth network mechanism 641 includes, for example, a recurrent neural network (RNN) architecture such as a long short-term memory (LSTM) or a gated recurrent unit (GRU). The fourth attention mechanism 642 includes a time series attention mechanism for determining which first part information I21 in the time series order should be paid attention to. Thus, the feature value of the first part information I21 considered to be the most important in the time series order is extracted and input to the combining unit 67.

[0091] The fifth network mechanism 651 extracts a feature value from the first part information I22 in the time series order. The fifth network mechanism 651 includes, for example, a recurrent neural network (RNN) architecture such as a long short-term memory (LSTM) or a gated recurrent unit (GRU). The fifth attention mechanism 652 includes a time series attention mechanism for determining which first part information I22 in the time series order should be paid attention to. Thus, the feature value of the first part information I22 considered to be the most important in the first part information I22 in the time series order is extracted and input into the combining unit 67.

[0092] The sixth network mechanism 66 extracts a feature value from the first part information I23 in the time series order. As described above, the first part information I23 is an image of the face (first part) of the mobile object 10. The sixth network mechanism 66 includes, for example, a three-dimensional convolutional neural network (3DCNN) or a combination of a convolutional neural network (CNN) and a recurrent neural network (RNN). Thus, the feature value is extracted from the first part information I23 in the time series order and input to the combining unit 67.

[0093] The combining unit 67 inputs the feature quantities of the overall information I11, I12, and I13 and the feature quantities of the part information (first part information) I21, I22, and I23 into the output unit 68. Fig.10 In this case, the combining unit 67 inputs the characteristic quantity of the overall information I11 considered to be the most important among the overall information I11 in the time series order from the first attention mechanism 612, the characteristic quantity of the overall information I12 considered to be the most important among the overall information I12 in the time series order from the second attention mechanism 622, the characteristic quantity of the overall information I13 in the time series order from the third network mechanism 63, the characteristic quantity of the first part information I21 considered to be the most important among the first part information I21 in the time series order from the fourth attention mechanism 642, the characteristic quantity of the first part information I22 considered to be the most important among the first part information I22 in the time series order from the fifth attention mechanism 652, and the characteristic quantity of the first part information I23 in the time series order from the sixth network mechanism 66 into the output unit 68.

[0094] The output unit 68 outputs the evaluation O2 of the moving direction of the moving object 10 based on the feature amount from the combining unit 67. The output unit 68 includes, for example, a full combining layer. The full combining layer includes, for example, a Softmax function.

[0095] like Figure 2As shown, when the resolution of the image of the moving object 10 detected from the target image information is medium resolution (S13; medium resolution), the calculation circuit 23 selects the medium resolution model 6 (detailed model, first model) from the plurality of learned models (S31), and uses the medium resolution model 6 for evaluation (S32). More specifically, the calculation circuit 23 inputs the overall information I11, I12, I13 and the part information (first part information) I21, I22, I23 of the moving object 10 detected from the target image information into the medium resolution model 6, and causes the medium resolution model 6 to output the evaluation O2 of the moving direction of the moving object 10 detected from the target image information. Thus, in the case of medium resolution, the evaluation system 2 evaluates the moving direction of the moving object 10 without using the information (second part information) with a high probability of low accuracy. As a result, the evaluation system 2 can improve the evaluation accuracy of the moving direction of the moving object 10.

[0096] Fig.11 7 is a schematic diagram illustrating an example of the configuration of the high-resolution model 7. Fig.11 In , the object image information includes a plurality of images G in a time series order. A bounding box B of the moving object 10 is detected from the plurality of images G. A plurality of overall information I11, I12, and I13 are obtained from each bounding box B. Thus, the overall information I11 in a time series order, the overall information I12 in a time series order, and the overall information I13 in a time series order are obtained. Fig.11 In FIG. 1 , the overall information I11 is the position of the moving object 10 , the overall information I12 is the speed of the moving object 10 , and the overall information I13 is an image of the entire moving object 10 .

[0097] exist Fig.11 In the example, a bounding box R of the face (first part) of the moving object 10 is detected from the plurality of images G. A plurality of first part information I21, I22, and I23 are obtained from each bounding box R. Thus, the first part information I21 in time series order, the first part information I22 in time series order, and the first part information I23 in time series order are obtained. Fig.11 In the figure, the first part information I21 is the position of the face of the moving object 10 , the first part information I22 is the orientation of the face of the moving object 10 , and the first part information I23 is the image of the face of the moving object 10 .

[0098] exist Fig.11 In , multiple skeleton points (second parts) P ​​of the moving object 10 are detected from multiple images G. The second part information I31 is obtained based on the correlation between the multiple skeleton points P. Thus, the second part information I31 in time series order is obtained. Fig.11In the example, the second part information I31 is information about the posture of the mobile body 10. In some examples, the second part information may also include information about a skeleton point, a position of a skeleton point, and adjacent information of a skeleton point.

[0099] Fig.11 The high-resolution model 7 outputs an evaluation O3 of the moving direction of the moving body 10 with respect to the input of the overall information I11 , I12 , I13 , the first part information I21 , I22 , I23 , and the second part information I31 . Fig.11 The high-resolution model 7 includes a first network mechanism 711, a first attention mechanism 712, a second network mechanism 721, a second attention mechanism 722, a third network mechanism 73, a fourth network mechanism 741, a fourth attention mechanism 742, a fifth network mechanism 751, a fifth attention mechanism 752, a sixth network mechanism 76, a seventh network mechanism 771, a seventh attention mechanism 772, a combining unit 78 and an output unit 79.

[0100] The first network mechanism 711, the first attention mechanism 712, the second network mechanism 721, the second attention mechanism 722 and the third network mechanism 73 are respectively Fig. 9 The first network mechanism 511, the first attention mechanism 512, the second network mechanism 521, the second attention mechanism 522 and the third network mechanism 53 of the low-resolution model 5 are the same.

[0101] The fourth network mechanism 741, the fourth attention mechanism 742, the fifth network mechanism 751, the fifth attention mechanism 752 and the sixth network mechanism 76 are respectively Fig.10 The fourth network mechanism 641, the fourth attention mechanism 642, the fifth network mechanism 651, the fifth attention mechanism 652 and the sixth network mechanism 66 of the medium-resolution model 6 are the same.

[0102] The seventh network mechanism 771 extracts a feature value from the second part information I31 in the time series order. The seventh network mechanism 771 includes, for example, a recurrent neural network (RNN) architecture such as a long short-term memory (LSTM) or a gated recurrent unit (GRU). The seventh attention mechanism 772 includes a time series attention mechanism for determining which second part information I31 in the time series order should be paid attention to. Thus, the feature value of the second part information I31 considered to be the most important in the second part information I31 in the time series order is extracted and input to the combining unit 78.

[0103] The combining unit 78 inputs the feature values ​​of the overall information I11, I12, and I13, the feature values ​​of the part information (first part information) I21, I22, and I23, and the feature value of the part information (second part information) I31 into the output unit 79. Fig.11In this case, the combining unit 78 inputs the characteristic quantity of the overall information I11 considered to be the most important among the overall information I11 in the time series order from the first attention mechanism 712, the characteristic quantity of the overall information I12 considered to be the most important among the overall information I12 in the time series order from the second attention mechanism 722, the characteristic quantity of the overall information I13 in the time series order from the third network mechanism 73, the characteristic quantity of the first part information I21 considered to be the most important among the first part information I21 in the time series order from the fourth attention mechanism 742, the characteristic quantity of the first part information I22 considered to be the most important among the first part information I22 in the time series order from the fifth attention mechanism 752, the characteristic quantity of the first part information I23 in the time series order from the sixth network mechanism 66, and the characteristic quantity of the second part information I31 considered to be the most important among the second part information I31 in the time series order from the seventh attention mechanism 772 into the output unit 79.

[0104] The output unit 79 outputs the evaluation O3 of the moving direction of the moving object 10 based on the feature amount from the combining unit 78. The output unit 79 includes, for example, a full combining layer. The full combining layer includes, for example, a Softmax function.

[0105] like Figure 2 As shown, when the resolution of the image of the moving object 10 detected from the target image information is high resolution (S13; high resolution), the calculation circuit 23 selects the high resolution model 7 (detailed model, second model) from the plurality of learned models (S41), and uses the high resolution model 7 for evaluation (S42). More specifically, the calculation circuit 23 inputs the overall information I11, I12, I13, the part information (first part information) I21, I22, I23, and the part information (second part information) I31 of the moving object 10 detected from the target image information into the high resolution model 7, and causes the high resolution model 7 to output the evaluation O3 of the moving direction of the moving object 10 detected from the target image information. Thus, in the case of high resolution, the evaluation system 2 increases the information and evaluates the moving direction of the moving object 10 compared to the case of medium resolution. As a result, the evaluation system 2 can improve the evaluation accuracy of the moving direction of the moving object 10.

[0106] The evaluation result of the moving direction of the moving object 10 by the evaluation system 2 described above can be used in the control system 4 of the moving device 1 to determine the operation of the moving device 1 .

[0107] Figure 12 to Figure 14 FIG. 1 is an explanatory diagram of an example of the operation of the mobile device 1. Fig.12 In FIG. 1 , there is a moving body 10 on the predetermined travel path M1 of the moving device 1. The moving body 10 is moving in the moving direction V0. Fig.12, the moving direction V0 is represented by the relative velocity vector of the moving body 10 with respect to the moving device 1 .

[0108] In the mobile device 1, the evaluation system 2 performs evaluation of the moving direction of the mobile body 10. The evaluation of the moving direction of the mobile body 10 includes the probability that the moving direction of the mobile body 10 is the first moving direction V1 and the probability that the moving direction of the mobile body 10 is the second moving direction V2. The first moving direction V1 is the direction in which the mobile body 10 is directed toward the object (mobile device 1). The second moving direction V2 is the direction in which the mobile body 10 is avoided from the object (mobile device 1).

[0109] When the probability that the moving direction of the moving object 10 is the first moving direction V1 is high, the moving object 10 is likely to reach the vicinity area 1a of the mobile device 1 and collide with the mobile device 1. Fig.13 As shown, in the mobile device 1 , the following actions can be taken: the control system 4 changes the planned travel path M1 of the mobile device 1 to a planned travel path M2 to avoid collision with the mobile body 10 .

[0110] When the probability that the moving direction of the moving object 10 is the second moving direction V2 is high, the possibility that the moving object 10 collides with the moving device 1 is low. Fig.14 As shown, in the mobile device 1 , an action is taken by the control system 4 to maintain the predetermined travel path M1 of the mobile device 1 .

[0111] In this way, in the mobile device 1, the control system 4 controls the action of the mobile device 1 according to the evaluation of the moving direction of the mobile body 10 obtained from the evaluation system 2, so that the possibility of the mobile device 1 colliding with the mobile body 10 can be reduced. In particular, the mobile device 1 can set a predetermined travel path by a yielding movement policy that takes into account whether the mobile body 10 takes an action to avoid collision with the mobile device 1, rather than a movement policy that gives priority to the movement of the mobile body 10, such as simply completely avoiding collision with the mobile body 10. As a result, the mobile device 1 can achieve smooth staggering with the mobile body 10.

[0112] [1.2 Effects, etc.]

[0113] The evaluation method described above is executed by a computing circuit 23 that can access a storage device 22 storing a plurality of learned models. The plurality of learned models include: a simple model (low-resolution model 5) that is learned in a manner that outputs an evaluation of the moving direction of the moving body 10 with respect to input of one or more overall information I11 to I13 based on the entirety of the moving body 10 to 13 (bounding boxes B, B1 to B3) in the image information; and a detailed model (medium-resolution model 6, high-resolution model 7) that is learned in a manner that outputs an evaluation of the moving direction of the moving body 10 with respect to input of at least one of the one or more overall information I11 to I13 and one or more part information I21 to I23, I31 based on one or more parts R, P of the moving body 10.

[0114] Here, in the evaluation method, it is confirmed that the resolution of the image of the moving object 10 to 13 detected from the target image information is less than a threshold value (first threshold value), and in response to the confirmation that the resolution is less than the threshold value (first threshold value), a simple model (low-resolution model 5) is selected from a plurality of learned models, and one or more pieces of overall information I11 to I13 of the moving object 10 detected from the target image information are input to the simple model (low-resolution model 5), so that the simple model (low-resolution model 5) outputs an evaluation O1 of the moving direction of the moving object 10 detected from the target image information. This configuration can improve the evaluation accuracy of the moving direction of the moving objects 10 to 13.

[0115] Alternatively, in the evaluation method, it is confirmed that the resolution of the image of the moving object 10 to 13 detected from the target image information is above a threshold value (first threshold value), and in response to confirming that the resolution is above the threshold value (first threshold value), a detailed model (medium resolution model 6, high resolution model 7) is selected from a plurality of learned models, and at least one of one or more overall information I11 to I13 and one or more part information I21 to I23, I31 of the moving object 10 to 13 detected from the target image information is input to the detailed model (medium resolution model 6, high resolution model 7), so that the detailed model (medium resolution model 6, high resolution model 7) outputs evaluations O2, O3 of the moving direction of the moving object 10 detected from the target image information. This configuration can improve the evaluation accuracy of the moving direction of the moving objects 10 to 13.

[0116] In the evaluation method, the resolution is determined based on the areas of the bounding boxes B, B1 to B3 of the moving objects 10 to 13 detected from the target image information. This configuration can improve the evaluation accuracy of the moving directions of the moving objects 10 to 13.

[0117] In the evaluation method, one or more pieces of overall information I11 to I13 include at least one of the positions of the moving objects 10 to 13, the speeds of the moving objects 10 to 13, and the images of the entire moving objects 10 to 13 (bounding boxes B, B1 to B3). This configuration can improve the evaluation accuracy of the moving directions of the moving objects 10 to 13.

[0118] In the evaluation method, one or more part information I21, I22, I23, I31 includes at least one of the positions of one or more parts R, P of the moving bodies 10 to 13, the orientations of one or more parts R, P of the moving bodies 10 to 13, images of one or more parts R, P of the moving bodies 10 to 13, and information based on the association between the multiple parts R, P of the moving bodies 10 to 13. This configuration can improve the evaluation accuracy of the moving directions of the moving bodies 10 to 13.

[0119] In the evaluation method, the positions of one or more parts R and P include the positions of the faces of the moving objects 10 to 13. This configuration can improve the evaluation accuracy of the moving directions of the moving objects 10 to 13.

[0120] In the evaluation method, the orientation of one or more parts R, P includes at least one of the orientation of the face of the moving objects 10 to 13 and the sight line of the moving objects 10 to 13. This configuration can improve the evaluation accuracy of the moving directions of the moving objects 10 to 13.

[0121] In the evaluation method, the images of one or more parts R, P include the images of the faces of the moving objects 10 to 13. This configuration can improve the evaluation accuracy of the moving directions of the moving objects 10 to 13.

[0122] In the evaluation method, information based on the correlation between the plurality of parts R and P includes information on the postures of the moving objects 10 to 13. This configuration can improve the evaluation accuracy of the moving directions of the moving objects 10 to 13.

[0123] In the evaluation method, the evaluation of the moving direction of the moving bodies 10 to 13 includes the probability that the moving direction of the moving bodies 10 to 13 is the first moving direction and the probability that the moving direction of the moving bodies 10 to 13 is the second moving direction. The first moving direction is the direction in which the moving bodies 10 to 13 are directed toward the object (moving device 1). The second moving direction is the direction in which the moving bodies 10 to 13 are away from the object (moving device 1). This configuration can realize efficient movement of the object taking into account the moving directions of the moving bodies 10 to 13.

[0124] From another point of view, the evaluation method described above is executed by a computing circuit 23 that can access a storage device 22 storing a plurality of learned models. The plurality of learned models include: a first model (medium resolution model 6) that is learned in a manner that outputs an evaluation O2 of the moving direction of the moving body 10 to 13 based on one or more overall information I11 to I13 of the entire moving body 10 to 13 (bounding boxes B, B1 to B3) and one or more first part information I21 to I23 based on one or more first parts R of the moving body 10 to 13 in the image information; and a second model (high resolution model 7) that is learned in a manner that outputs an evaluation O3 of the moving direction of the moving body 10 to 13 based on at least one of the one or more overall information I11 to I13, at least one of the one or more first part information I21 to I23, and one or more second part information I31 based on one or more second parts P of the moving body 10 to 13. The one or more second parts P are smaller than the one or more first parts R.

[0125] Here, in the evaluation method, it is confirmed that the resolution of the image of the moving object 10 to 13 detected from the target image information is less than a threshold value (second threshold value), and in response to the confirmation that the resolution is less than the threshold value (second threshold value), a first model (medium-resolution model 6) is selected from a plurality of learned models, and one or more pieces of overall information I11 to I13 and one or more pieces of first part information I21 to I23 of the moving object 10 to 13 detected from the target image information are input to the first model (medium-resolution model 6), so that the first model (medium-resolution model 6) outputs an evaluation O2 of the moving direction of the moving object 10 to 13 detected from the target image information. This configuration can improve the evaluation accuracy of the moving direction of the moving objects 10 to 13.

[0126] Alternatively, in the evaluation method, it is confirmed that the resolution of the image of the moving object 10 to 13 detected from the target image information is above a threshold value (second threshold value), and in response to confirming that the distance D1 to D3 is less than or equal to the threshold value, a second model (high-resolution model 7) is selected from a plurality of learned models, at least one of the one or more overall information I11 to I13 of the moving object 10 to 13 detected from the target image information, at least one of the one or more first part information I21 to I23, and one or more second part information I31 are input to the second model (high-resolution model 7), and the second model (high-resolution model 7) outputs the evaluation O3 of the moving direction of the moving object 10 to 13 detected from the target image information. This configuration can improve the evaluation accuracy of the moving direction of the moving objects 10 to 13.

[0127] The program described above is a program for causing the calculation circuit 23 to execute the above-mentioned evaluation method. This configuration can improve the evaluation accuracy of the moving directions of the moving bodies 10 to 13.

[0128] The evaluation system 2 described above includes: a storage device 22 storing a plurality of learned models, and a computing circuit 23 capable of accessing the storage device 22. The plurality of learned models include: a simple model (low-resolution model 5) learned in a manner that outputs an evaluation of the moving direction of the moving body 10 with respect to input of one or more overall information I11 to I13 based on the entirety of the moving body 10 to 13 (bounding boxes B, B1 to B3) in the image information; and a detailed model (medium-resolution model 6, high-resolution model 7) learned in a manner that outputs an evaluation of the moving direction of the moving body 10 with respect to input of at least one of the one or more overall information I11 to I13 and one or more part information I21 to I23, I31 based on one or more parts R, P of the moving body 10. When the resolution of the image of the moving object 10 to 13 detected from the target image information is less than a threshold value (first threshold value), the operation circuit 23 selects a simple model (low-resolution model 5) from the plurality of learned models, and inputs one or more pieces of overall information I11 to I13 of the moving object 10 to 13 detected from the target image information into the simple model (low-resolution model 5), so that the simple model (low-resolution model 5) outputs the evaluation O1 of the moving direction of the moving object 10 detected from the target image information. When the resolution is equal to or greater than the threshold value (first threshold value), the operation circuit 23 selects a detailed model (medium-resolution model 6, high-resolution model 7) from the plurality of learned models, and inputs at least one of the one or more pieces of overall information I11 to I13 of the moving object 10 detected from the target image information and one or more pieces of part information I21 to I23, I31 into the detailed model (medium-resolution model 6, high-resolution model 7), so that the detailed model (medium-resolution model 6, high-resolution model 7) outputs the evaluation O2, O3 of the moving direction of the moving object 10 to 13 detected from the target image information. This configuration can improve the accuracy of evaluating the moving directions of the moving bodies 10 to 13 .

[0129] The evaluation system 2 described above includes a storage device 22 storing a plurality of learned models and a computing circuit 23 capable of accessing the storage device 22. The plurality of learned models include a first model (medium resolution model 6) learned in a manner to output an evaluation O2 of the moving direction of the moving body 10 to 13 based on one or more overall information I11 to I13 of the entire moving body 10 to 13 (bounding boxes B, B1 to B3) and one or more first part information I21 to I23 based on one or more first parts R of the moving body 10 to 13 in the image information; and a second model (high resolution model 7) learned in a manner to output an evaluation O3 of the moving direction of the moving body 10 to 13 based on at least one of the one or more overall information I11 to I13, at least one of the one or more first part information I21 to I23, and one or more second part information I31 based on one or more second parts P of the moving body 10 to 13. The one or more second parts P are smaller than the one or more first parts R. When the resolution of the image of the moving objects 10 to 13 detected from the target image information is less than a threshold value (second threshold value), the operation circuit 23 selects a first model (medium resolution model 6) from the plurality of learned models, and inputs one or more pieces of overall information I11 to I13 and one or more pieces of first part information I21 to I23 of the moving objects 10 to 13 detected from the target image information into the first model (medium resolution model 6), so that the first model (medium resolution model 6) outputs an evaluation O2 of the moving direction of the moving objects 10 to 13 detected from the target image information. When the resolution is equal to or greater than the threshold value (second threshold value), the operation circuit 23 selects a second model (high resolution model 7) from the plurality of learned models, and inputs at least one of the one or more pieces of overall information I11 to I13, at least one of the one or more pieces of first part information I21 to I23, and one or more pieces of second part information I31 of the moving objects 10 to 13 detected from the target image information into the second model (high resolution model 7), so that the second model (high resolution model 7) outputs an evaluation O3 of the moving direction of the moving objects 10 to 13 detected from the target image information. This configuration can improve the accuracy of evaluating the moving directions of the moving bodies 10 to 13 .

[0130] [2. Modifications]

[0131] The embodiments of the present disclosure are not limited to the above embodiments. As long as the above embodiments can achieve the technical problems of the present disclosure, various changes can be made according to the design, etc. The following lists the modified examples of the above embodiments. The modified examples described below can be appropriately combined and applied.

[0132] In a modified example, the plurality of learned models may include two of the low-resolution model 5, the medium-resolution model 6, and the high-resolution model 7 instead of all of them. For example, the operation circuit 23 may select the low-resolution model 5 when the resolution of the image of the moving object 10 detected from the object image information is less than a threshold value, and may select the medium-resolution model 6 or the high-resolution model 7 when the resolution of the image of the moving object 10 detected from the object image information is greater than the threshold value. For example, the operation circuit 23 may select the medium-resolution model 6 when the resolution of the image of the moving object 10 detected from the object image information is less than a threshold value, and may select the high-resolution model 7 when the resolution of the image of the moving object 10 detected from the object image information is greater than the threshold value.

[0133] In a modified example, the simple model used as the low-resolution model 5 can be learned in a manner that outputs an evaluation of the moving direction of the moving object 10 in response to input of one or more pieces of overall information I11 to I13 based on the entire moving object 10 in the image information. In the simple model, the amount of overall information is not particularly limited.

[0134] In a modified example, the detailed model used as the medium-resolution model 6 or the high-resolution model 7 can be learned in a manner that outputs an evaluation of the moving direction of the moving body 10 with respect to at least one of one or more pieces of overall information I11 to I13 based on the entirety of the moving body 10 in the image information and one or more pieces of part information I21 to I23 based on one or more parts of the moving body 10 in the image information. In the detailed model, the number of pieces of overall information and the number of pieces of part information are not particularly limited. In particular, the detailed model does not necessarily have to use all of the overall information used in the simple model.

[0135] In a modified example, the first model used as the medium-resolution model 6 can be learned in a manner that outputs an evaluation of the moving direction of the moving object 10 with respect to input of one or more pieces of overall information I11 to I13 based on the entirety of the moving object 10 in the image information and one or more pieces of first part information I21 to I23 based on one or more first parts of the moving object 10 in the image information. In the first model, the number of pieces of overall information and the number of pieces of first part information are not particularly limited.

[0136] In a modified example, the second model used as the high-resolution model 7 can be learned in a manner that outputs an evaluation of the moving direction of the moving body 10 with respect to at least one of the one or more overall information I11 to I13 based on the entire moving body 10 in the image information, at least one of the one or more first part information I21 to I23 based on one or more first parts of the moving body 10 in the image information, and at least one of the one or more second part information I31 based on one or more second parts of the moving body 10 in the image information. In the second model, the number of overall information, the number of first part information, and the number of second part information are not particularly limited. In particular, the second model does not necessarily have to use all the overall information and all the first part information used in the first model.

[0137] In a modified example, the interface 21 of the evaluation system 2 does not need to include both an input / output device and a communication device.

[0138] In a modified example, the imaging system 3 may include a plurality of cameras having different performances (resolutions). The evaluation system 2 may evaluate the moving direction of the moving object with respect to the target image information obtained from the plurality of cameras having different performances (resolutions).

[0139] In a modified example, the evaluation system 2 does not have to be mounted on the mobile device 1. The evaluation system 2 can be used in devices or systems other than the mobile device 1. For example, the evaluation system 2 can be used in an alarm system that issues a warning of a collision with the mobile body 10. The evaluation system 2 can also be implemented by a computer system such as multiple servers. In other words, the multiple functions (constituent elements) in the evaluation system 2 do not have to be integrated in one housing, and the constituent elements of the evaluation system 2 can also be dispersed in multiple housings. Furthermore, at least a part of the functions of the evaluation system 2, such as a part of the functions of the computing circuit 23, can also be implemented through the cloud (cloud computing) or the like.

[0140] [3. Method]

[0141] As is clear from the above-mentioned embodiments and variations, the present disclosure includes the following aspects. Below, the reference numerals are marked in brackets only to indicate the correspondence with the embodiments. In addition, in consideration of the ease of reading the sentences, the second and subsequent reference numerals in brackets are sometimes omitted.

[0142] The first method is an evaluation method executed by a computing circuit (23) capable of accessing a storage device (22) storing a plurality of learned models. The plurality of learned models include: a simple model (low-resolution model 5) learned in a manner that outputs an evaluation of the moving direction of the moving object (10) with respect to input of one or more overall information (I11 to I13) based on the entirety of the moving object (10 to 13) (bounding boxes B, B1 to B3) in image information; and a detailed model (medium-resolution model 6, high-resolution model 7) learned in a manner that outputs an evaluation of the moving direction of the moving object (10) with respect to input of at least one of the one or more overall information (I11 to I13) and one or more part information (I21 to I23, I31) based on one or more parts (R, P) of the moving object (10). In the evaluation method, when the resolution of the image of the moving object (10~13) detected from the object image information is less than a threshold value (first threshold value), the simple model (low-resolution model 5) is selected from the multiple learned models, and the one or more overall information (I11~I13) of the moving object (10~13) detected from the object image information is input into the simple model (low-resolution model 5), so that the simple model (low-resolution model 5) outputs an evaluation (O1) of the moving direction of the moving object (10) detected from the object image information. The evaluation method selects the detailed model (medium resolution model 6, high resolution model 7) from the plurality of learned models when the resolution is greater than the threshold value (first threshold value), and inputs at least one of the one or more overall information (I11 to I13) and the one or more part information (I21 to I23, I31) of the moving object (10) detected from the target image information into the detailed model (medium resolution model 6, high resolution model 7), so that the detailed model (medium resolution model 6, high resolution model 7) outputs an evaluation (O2, O3) of the moving direction of the moving object (10 to 13) detected from the target image information. This method can improve the evaluation accuracy of the moving direction of the moving object (10 to 13).

[0143] The second method is an evaluation method executed by a computing circuit (23) capable of accessing a storage device (22) storing a plurality of learned models. The plurality of learned models include: a simple model (low-resolution model 5) learned in a manner that outputs an evaluation of the moving direction of the moving object (10) with respect to input of one or more overall information (I11 to I13) based on the entirety of the moving object (10 to 13) (bounding boxes B, B1 to B3) in image information; and a detailed model (medium-resolution model 6, high-resolution model 7) learned in a manner that outputs an evaluation of the moving direction of the moving object (10) with respect to input of at least one of the one or more overall information (I11 to I13) and one or more part information (I21 to I23, I31) based on one or more parts (R, P) of the moving object (10). In the evaluation method, it is confirmed that the resolution of the image of the moving object (10-13) detected from the object image information is less than a threshold value (first threshold value), and in response to confirming that the resolution is less than the threshold value (first threshold value), the simple model (low-resolution model 5) is selected from the plurality of learned models, and the one or more overall information (I11-I13) of the moving object (10) detected from the object image information is input into the simple model (low-resolution model 5), so that the simple model (low-resolution model 5) outputs an evaluation (O1) of the moving direction of the moving object (10) detected from the object image information. This method can improve the evaluation accuracy of the moving direction of the moving object (10-13).

[0144] A third method is an evaluation method executed by a computing circuit (23) capable of accessing a storage device (22) storing a plurality of learned models. The plurality of learned models include: a simple model (low-resolution model 5) learned in a manner that outputs an evaluation of the moving direction of the moving object (10) with respect to input of one or more overall information (I11 to I13) based on the entirety of the moving object (10 to 13) (bounding boxes B, B1 to B3) in image information; and a detailed model (medium-resolution model 6, high-resolution model 7) learned in a manner that outputs an evaluation of the moving direction of the moving object (10) with respect to input of at least one of the one or more overall information (I11 to I13) and one or more part information (I21 to I23, I31) based on one or more parts (R, P) of the moving object (10). In the evaluation method, it is confirmed that the resolution of the image of the moving object (10-13) detected from the target image information is above a threshold value (first threshold value), and in response to confirming that the resolution is above the threshold value (first threshold value), the detailed model (medium resolution model 6, high resolution model 7) is selected from the plurality of learned models, at least one of the one or more overall information (I11-I13) and the one or more part information (I21-I23, I31) of the moving object (10-13) detected from the target image information is input into the detailed model (medium resolution model 6, high resolution model 7), so that the detailed model (medium resolution model 6, high resolution model 7) outputs an evaluation (O2, O3) of the moving direction of the moving object (10) detected from the target image information. This method can improve the evaluation accuracy of the moving direction of the moving object (10-13).

[0145] A fourth aspect is an evaluation method based on any one of the first to third aspects. In the fourth aspect, the resolution is determined based on the area of ​​a bounding box (B, B1 to B3) of the moving object (10 to 13) detected from the object image information. This method does not require actual measurement of the distance (D1 to D3), and therefore can reduce the cost of equipment required to implement the evaluation method.

[0146] A fifth aspect is an evaluation method based on any one of the first to fourth aspects. In the fifth aspect, the one or more overall information (I11 to I13) includes at least one of the position of the moving object (10 to 13), the speed of the moving object (10 to 13), and an image of the entire moving object (10 to 13) (bounding box B, B1 to B3). This aspect can improve the evaluation accuracy of the moving direction of the moving object (10 to 13).

[0147] A sixth aspect is an evaluation method based on any one of the first to fifth aspects. In the sixth aspect, the one or more part information (I21, I22, I23, I31) includes at least one of the positions of the one or more parts (R, P) of the moving body (10-13), the orientations of the one or more parts (R, P) of the moving body (10-13), images of the one or more parts (R, P) of the moving body (10-13), and information based on the association of the multiple parts (R, P) of the moving body (10-13). This aspect can improve the evaluation accuracy of the moving direction of the moving body (10-13).

[0148] The seventh aspect is an evaluation method based on the sixth aspect. In the seventh aspect, the positions of the one or more parts (R, P) include the position of the face of the moving object (10-13). This aspect can improve the evaluation accuracy of the moving direction of the moving object (10-13).

[0149] The eighth aspect is an evaluation method based on the sixth or seventh aspect. In the eighth aspect, the orientation of the one or more parts (R, P) includes at least one of the orientation of the face of the moving object (10-13) and the sight line of the moving object (10-13). This aspect can improve the evaluation accuracy of the moving direction of the moving object (10-13).

[0150] A ninth aspect is an evaluation method based on any one of the sixth to eighth aspects. In the ninth aspect, the image of the one or more parts (R, P) includes an image of the face of the moving object (10-13). This aspect can improve the evaluation accuracy of the moving direction of the moving object (10-13).

[0151] The tenth aspect is an evaluation method based on any one of the sixth to ninth aspects. In the tenth aspect, the information based on the correlation between the plurality of parts (R, P) includes information related to the posture of the moving body (10-13). This aspect can improve the evaluation accuracy of the moving direction of the moving body (10-13).

[0152] The eleventh method is an evaluation method based on any one of the first to tenth methods. In the tenth method, the evaluation of the moving direction of the moving body (10-13) includes the probability that the moving direction of the moving body (10-13) is a first moving direction and the probability that the moving direction of the moving body (10-13) is a second moving direction. The first moving direction is the direction of the moving body (10-13) toward the object (moving device 1). The second moving direction is the direction of the moving body (10-13) avoiding the object (moving device 1). This method can realize efficient movement of the object taking into account the moving direction of the moving body (10-13).

[0153] The twelfth aspect is an evaluation method executed by a calculation circuit (23) capable of accessing a storage device (22) storing a plurality of learned models. The plurality of learned models include: a first model (medium resolution model 6) learned in a manner to output an evaluation (O2) of a moving direction of the moving object (10-13) based on one or more overall information (I11-I13) of the moving object (10-13) as a whole (bounding box B, B1-B3) in image information and one or more first part information (I21-I23) based on one or more first parts (R) of the moving object (10-13); and a second model (high resolution model 7) learned in a manner to output an evaluation (O3) of a moving direction of the moving object (10-13) based on at least one of the one or more overall information (I11-I13), at least one of the one or more first part information (I21-I23), and one or more second part information (I31) based on one or more second parts (P) of the moving object (10-13). The one or more second parts (P) are smaller than the one or more first parts (R). In the evaluation method, when the resolution of the image of the moving body (10~13) detected from the object image information is less than a threshold value (second threshold value), the first model (medium-resolution model 6) is selected from the multiple learned models, and the one or more overall information (I11~I13) and the one or more first part information (I21~I23) of the moving body (10~13) detected from the object image information are input into the first model (medium-resolution model 6), so that the first model (medium-resolution model 6) outputs an evaluation (O2) of the moving direction of the moving body (10~13) detected from the object image information. The evaluation method selects the second model (high-resolution model 7) from the plurality of learned models when the resolution is greater than the threshold (second threshold), and inputs at least one of the one or more overall information (I11 to I13) of the moving object (10 to 13) detected from the target image information, at least one of the one or more first part information (I21 to I23), and the one or more second part information (I31) into the second model (high-resolution model 7), so that the second model (high-resolution model 7) outputs an evaluation (O3) of the moving direction of the moving object (10 to 13) detected from the target image information. This method can improve the evaluation accuracy of the moving direction of the moving object (10 to 13).

[0154] The thirteenth aspect is an evaluation method executed by a calculation circuit (23) capable of accessing a storage device (22) storing a plurality of learned models. The plurality of learned models include: a first model (medium resolution model 6) learned in a manner to output an evaluation (O2) of a moving direction of the moving object (10-13) based on one or more overall information (I11-I13) of the moving object (10-13) as a whole (bounding box B, B1-B3) in image information and one or more first part information (I21-I23) based on one or more first parts (R) of the moving object (10-13); and a second model (high resolution model 7) learned in a manner to output an evaluation (O3) of a moving direction of the moving object (10-13) based on at least one of the one or more overall information (I11-I13), at least one of the one or more first part information (I21-I23), and one or more second part information (I31) based on one or more second parts (P) of the moving object (10-13). The one or more second parts (P) are smaller than the one or more first parts (R). In the evaluation method, it is confirmed that the resolution of the image of the moving object (10-13) detected from the object image information is less than a threshold value (second threshold value), and in response to confirming that the resolution is less than the threshold value (second threshold value), the first model (medium resolution model 6) is selected from the plurality of learned models, and the one or more overall information (I11-I13) and the one or more first part information (I21-I23) of the moving object (10-13) detected from the object image information are input into the first model (medium resolution model 6), so that the first model (medium resolution model 6) outputs an evaluation (O2) of the moving direction of the moving object (10-13) detected from the object image information. This method can improve the evaluation accuracy of the moving direction of the moving object (10-13).

[0155] A fourteenth aspect is an evaluation method executed by a calculation circuit (23) capable of accessing a storage device (22) storing a plurality of learned models. The plurality of learned models include: a first model (medium resolution model 6) learned in a manner to output an evaluation (O2) of a moving direction of the moving object (10-13) based on one or more overall information (I11-I13) of the moving object (10-13) as a whole (bounding box B, B1-B3) in image information and one or more first part information (I21-I23) based on one or more first parts (R) of the moving object (10-13); and a second model (high resolution model 7) learned in a manner to output an evaluation (O3) of a moving direction of the moving object (10-13) based on at least one of the one or more overall information (I11-I13), at least one of the one or more first part information (I21-I23), and one or more second part information (I31) based on one or more second parts (P) of the moving object (10-13). The one or more second parts (P) are smaller than the one or more first parts (R). In the evaluation method, it is confirmed that the resolution of the image of the moving object (10-13) detected from the target image information is greater than a threshold (second threshold), and in response to confirming that the resolution is greater than the second threshold, the second model (high-resolution model 7) is selected from the plurality of learned models, at least one of the one or more overall information (I11-I13) of the moving object (10-13) detected from the target image information, at least one of the one or more first part information (I21-I23), and the one or more second part information (I31) are input into the second model (high-resolution model 7), so that the second model (high-resolution model 7) outputs an evaluation (O3) of the moving direction of the moving object (10-13) detected from the target image information. This method can improve the evaluation accuracy of the moving direction of the moving object (10-13).

[0156] A fifteenth aspect is a program for causing the arithmetic circuit (23) to execute an evaluation method according to any one of the first to fourteenth aspects. This aspect can improve the evaluation accuracy of the moving direction of the moving body (10 to 13).

[0157] The evaluation system (2) of the sixteenth mode comprises: a storage device (22) storing a plurality of learned models, and a computing circuit (23) capable of accessing the storage device (22). The plurality of learned models include: a simple model (low-resolution model 5) learned in a manner that outputs an evaluation of the moving direction of the moving object (10) with respect to input of one or more overall information (I11 to I13) based on the entirety of the moving object (10 to 13) (bounding boxes B, B1 to B3) in image information; and a detailed model (medium-resolution model 6, high-resolution model 7) learned in a manner that outputs an evaluation of the moving direction of the moving object (10) with respect to input of at least one of the one or more overall information (I11 to I13) and one or more part information (I21 to I23, I31) based on one or more parts (R, P) of the moving object (10). When the resolution of the image of the moving body (10-13) detected from the object image information is less than a threshold value (first threshold value), the operation circuit (23) selects the simple model (low-resolution model 5) from the multiple learned models, and inputs the one or more overall information (I11-I13) of the moving body (10-13) detected from the object image information into the simple model (low-resolution model 5), so that the simple model (low-resolution model 5) outputs an evaluation (O1) of the moving direction of the moving body (10) detected from the object image information. When the resolution is greater than the threshold value (first threshold value), the operation circuit (23) selects the detailed model (medium resolution model 6, high resolution model 7) from the plurality of learned models, and inputs at least one of the one or more overall information (I11 to I13) and the one or more part information (I21 to I23, I31) of the moving object (10) detected from the target image information into the detailed model (medium resolution model 6, high resolution model 7), so that the detailed model (medium resolution model 6, high resolution model 7) outputs an evaluation (O2, O3) of the moving direction of the moving object (10 to 13) detected from the target image information. This method can improve the evaluation accuracy of the moving direction of the moving object (10 to 13).

[0158] An evaluation system (2) according to a seventeenth aspect includes: a storage device (22) storing a plurality of learned models; and a calculation circuit (23) capable of accessing the storage device (22). The plurality of learned models include: a first model (medium resolution model 6) learned in a manner to output an evaluation (O2) of a moving direction of the moving object (10-13) based on one or more overall information (I11-I13) of the moving object (10-13) as a whole (bounding box B, B1-B3) in image information and one or more first part information (I21-I23) based on one or more first parts (R) of the moving object (10-13); and a second model (high resolution model 7) learned in a manner to output an evaluation (O3) of a moving direction of the moving object (10-13) based on at least one of the one or more overall information (I11-I13), at least one of the one or more first part information (I21-I23), and one or more second part information (I31) based on one or more second parts (P) of the moving object (10-13). The one or more second parts (P) are smaller than the one or more first parts (R). When the resolution of the image of the moving body (10-13) detected from the object image information is less than a threshold value (second threshold value), the operation circuit (23) selects the first model (medium-resolution model 6) from the multiple learned models, and inputs the one or more overall information (I11-I13) and the one or more first part information (I21-I23) of the moving body (10-13) detected from the object image information into the first model (medium-resolution model 6), so that the first model (medium-resolution model 6) outputs an evaluation (O2) of the moving direction of the moving body (10-13) detected from the object image information. When the resolution is greater than or equal to the threshold value (second threshold value), the operation circuit (23) selects the second model (high-resolution model 7) from the plurality of learned models, and inputs at least one of the one or more overall information (I11 to I13) of the moving object (10 to 13) detected from the target image information, at least one of the one or more first part information (I21 to I23), and the one or more second part information (I31) into the second model (high-resolution model 7), so that the second model (high-resolution model 7) outputs an evaluation (O3) of the moving direction of the moving object (10 to 13) detected from the target image information. This method can improve the evaluation accuracy of the moving direction of the moving object (10 to 13).

[0159] Furthermore, the second to eleventh aspects can also be appropriately modified and applied to the sixteenth or seventeenth aspect.

[0160] [4. Terminology]

[0161] In this disclosure, terms related to machine learning are defined and used as follows.

[0162] A “learned model” is an “inference program” that embeds “learned parameters”.

[0163] "Learned parameters" refer to parameters (coefficients) obtained as a result of learning using a learning data set. Learned parameters are generated by inputting a learning data set into a learning program and mechanically adjusting it for a certain purpose. Although learned parameters are adjusted according to the purpose of learning, the individual body is just a simple parameter (information such as a numerical value), and it is embedded in the inference program to function as a learned model for the first time. For example, in the case of deep learning, parameters used to weight the links between nodes correspond to the main parameters in the learned parameters.

[0164] An "inference program" is a program that can output a certain result relative to an input by applying embedded learned parameters. For example, it is a program that specifies a series of calculation steps for applying learned parameters obtained as a result of learning to an image provided as input and outputting a result (authentication, judgment) relative to the image.

[0165] "Learning data set" is also called training data set, which refers to secondary processed data generated by preprocessing the original data such as removing missing values ​​and outliers, adding other data such as label information (correct answer data), or combining them and performing conversion / processing to facilitate the analysis of the method based on the object of learning. The learning data set sometimes also includes data that has been so-called "expanded" by applying certain transformations to the original data.

[0166] “Raw data” refers to data that is acquired once by users, suppliers, other operators, research institutions, etc., and that has been converted and processed in a way that can be read into a database.

[0167] The “learning program” is a program that executes an algorithm for finding a certain rule from a learning data set and generating a model that expresses the rule. Specifically, it is a program that specifies steps to be executed by a computer in order to realize learning based on an adopted learning method.

[0168] (Industrial Applicability)

[0169] The present disclosure can be applied to an evaluation method, a program (computer program), and an evaluation system. Specifically, the present disclosure can be applied to an evaluation method, a program (computer program), and an evaluation system related to evaluation of the moving direction of a moving object captured in an image.

[0170] Description of Reference Numerals

[0171] 1: moving device (object), 2: evaluation system, 22: storage device, 23: calculation circuit, 5: low-resolution model (simple model), 6: medium-resolution model (detailed model, first model), 7: high-resolution model (detailed model, second model), 10, 11 to 13: moving body, B, B1 to B3: bounding box, R: part (first part), P: part (second part), D1 to D3: distance, I11 to I13: overall information, I21 to I23: part information (first part information), I31: part information (second part information), O1 to O3: evaluation, V1: first moving direction, V2: second moving direction.

Claims

1. An evaluation method, performed by a computing circuit capable of accessing a storage device storing a plurality of learned models, The multiple learned models include: The simple model is learned in a manner that outputs an evaluation of the moving direction of the moving object in response to input of one or more pieces of overall information based on the entire moving object in the image information; as well as a detailed model that is learned in a manner that outputs an evaluation of the moving direction of the moving body with respect to at least one of the one or more overall information and one or more part information of one or more parts of the moving body, In the evaluation method, When the resolution of the image of the moving object detected from the object image information is less than a threshold value, the simple model is selected from the multiple learned models, and the one or more overall information of the moving object detected from the object image information is input into the simple model, so that the simple model outputs an evaluation of the moving direction of the moving object detected from the object image information, When the resolution is above the threshold, the detailed model is selected from the multiple learned models, and at least one of the one or more overall information and the one or more part information of the moving body detected from the object image information are input into the detailed model, so that the detailed model outputs an evaluation of the moving direction of the moving body detected from the object image information.

2. An evaluation method, performed by a computing circuit capable of accessing a storage device storing a plurality of learned models, The multiple learned models include: The simple model is learned in a manner that outputs an evaluation of the moving direction of the moving object in response to input of one or more pieces of overall information based on the entire moving object in the image information; as well as a detailed model that is learned in a manner that outputs an evaluation of the moving direction of the moving body with respect to at least one of the one or more overall information and one or more part information of one or more parts of the moving body, In the evaluation method, confirming that the resolution of the image of the moving object detected from the object image information is less than a threshold value, In response to determining that the resolution is less than the threshold, selecting the simplified model from the plurality of learned models, The one or more pieces of overall information of the moving object detected from the object image information are input to the simple model, so that the simple model outputs an evaluation of the moving direction of the moving object detected from the object image information.

3. An evaluation method, performed by a computing circuit capable of accessing a storage device storing a plurality of learned models, The multiple learned models include: The simple model is learned in a manner that outputs an evaluation of the moving direction of the moving object in response to input of one or more pieces of overall information based on the entire moving object in the image information; as well as a detailed model that is learned in a manner that outputs an evaluation of the moving direction of the moving body with respect to at least one of the one or more overall information and one or more part information of one or more parts of the moving body, In the evaluation method, confirming that the resolution of the image of the moving object detected from the target image information is greater than a threshold value, In response to confirming that the resolution is greater than or equal to the threshold, selecting the detailed model from the plurality of learned models, At least one of the one or more overall information of the moving object detected from the object image information and the one or more part information are input into the detailed model, so that the detailed model outputs an evaluation of the moving direction of the moving object detected from the object image information.

4. The evaluation method according to any one of claims 1 to 3, wherein: The resolution is determined based on the area of ​​a bounding box of the moving object detected from the target image information.

5. The evaluation method according to any one of claims 1 to 3, wherein: The one or more pieces of overall information include at least one of a position of the moving object, a speed of the moving object, and an image of the entire moving object.

6. The evaluation method according to any one of claims 1 to 3, wherein: The one or more part information includes at least one of the positions of the one or more parts of the moving body, the orientations of the one or more parts of the moving body, images of the one or more parts of the moving body, and information of the moving body based on the association between the multiple parts.

7. The evaluation method according to claim 6, wherein: The positions of the one or more parts include the position of the face of the moving object.

8. The evaluation method according to claim 6, wherein: The orientation of the one or more parts includes at least one of the orientation of the face of the moving object and the line of sight of the moving object.

9. The evaluation method according to claim 6, wherein: The one or more images of the parts include an image of a face of the moving object.

10. The evaluation method according to claim 6, wherein: The information based on the association between the plurality of parts includes information related to the posture of the moving object.

11. The evaluation method according to any one of claims 1 to 3, wherein: The evaluation of the moving direction of the moving object includes a probability that the moving direction of the moving object is a first moving direction and a probability that the moving direction of the moving object is a second moving direction. The first moving direction is the direction of the moving body toward the object. The second moving direction is a direction in which the moving body avoids the object.

12. An evaluation method, performed by a computing circuit capable of accessing a storage device storing a plurality of learned models, The multiple learned models include: The first model is learned in a manner to output an evaluation of the moving direction of the moving object with respect to input of one or more pieces of overall information based on the entire moving object and one or more pieces of first part information based on one or more first parts of the moving object in the image information; as well as The second model is learned in a manner that outputs an evaluation of the moving direction of the moving body relative to at least one of the one or more overall information, at least one of the one or more first part information, and one or more second part information based on the input of one or more second parts of the moving body, The one or more second portions are smaller than the one or more first portions, In the evaluation method, When the resolution of the image of the moving object detected from the object image information is less than a threshold value, the first model is selected from the multiple learned models, and the one or more overall information and the one or more first part information of the moving object detected from the object image information are input into the first model, so that the first model outputs an evaluation of the moving direction of the moving object detected from the object image information, When the resolution is above the threshold, the second model is selected from the multiple learned models, and at least one of the one or more overall information of the moving body detected from the object image information, at least one of the one or more first part information, and the one or more second part information are input into the second model, so that the second model outputs an evaluation of the moving direction of the moving body detected from the object image information.

13. An evaluation method, performed by a computing circuit capable of accessing a storage device storing a plurality of learned models, The multiple learned models include: The first model is learned in a manner to output an evaluation of the moving direction of the moving object with respect to input of one or more pieces of overall information based on the entire moving object and one or more pieces of first part information based on one or more first parts of the moving object in the image information; as well as The second model is learned in a manner that outputs an evaluation of the moving direction of the moving body relative to at least one of the one or more overall information, at least one of the one or more first part information, and one or more second part information based on the input of one or more second parts of the moving body, The one or more second portions are smaller than the one or more first portions, In the evaluation method, confirming that the resolution of the image of the moving object detected from the object image information is less than a threshold value, selecting the first model from the plurality of learned models in response to determining that the resolution is less than the threshold, The one or more pieces of overall information and the one or more pieces of first part information of the moving object detected from the object image information are input into the first model, so that the first model outputs an evaluation of the moving direction of the moving object detected from the object image information.

14. An evaluation method, performed by a computing circuit capable of accessing a storage device storing a plurality of learned models, The multiple learned models include: The first model is learned in a manner to output an evaluation of the moving direction of the moving object with respect to input of one or more pieces of overall information based on the entire moving object and one or more pieces of first part information based on one or more first parts of the moving object in the image information; as well as The second model is learned in a manner that outputs an evaluation of the moving direction of the moving body relative to at least one of the one or more overall information, at least one of the one or more first part information, and one or more second part information based on the input of one or more second parts of the moving body, The one or more second portions are smaller than the one or more first portions, In the evaluation method, confirming that the resolution of the image of the moving object detected from the target image information is greater than a threshold value, In response to confirming that the resolution is greater than the threshold, selecting the second model from the plurality of learned models, At least one of the one or more overall information of the moving body detected from the object image information, at least one of the one or more first part information, and the one or more second part information are input into the second model, so that the second model outputs an evaluation of the moving direction of the moving body detected from the object image information. 15 . A program for causing a calculation circuit to execute the evaluation method according to claim 1 .

16. An evaluation system comprising: A storage device for storing a plurality of learned models; as well as an arithmetic circuit capable of accessing the storage device, The multiple learned models include: The simple model is learned in a manner that outputs an evaluation of the moving direction of the moving object in response to input of one or more pieces of overall information based on the entire moving object in the image information; as well as a detailed model that is learned in a manner that outputs an evaluation of the moving direction of the moving body with respect to at least one of the one or more overall information and one or more part information of one or more parts of the moving body, When the resolution of the image of the moving object detected from the object image information is less than a threshold value, the operation circuit selects the simple model from the multiple learned models, and inputs the one or more overall information of the moving object detected from the object image information into the simple model, so that the simple model outputs an evaluation of the moving direction of the moving object detected from the object image information. When the resolution is above the threshold, the operation circuit selects the detailed model from the multiple learned models, and inputs at least one of the one or more overall information and the one or more part information of the moving body detected from the object image information into the detailed model, so that the detailed model outputs an evaluation of the moving direction of the moving body detected from the object image information.

17. An evaluation system comprising: A storage device for storing a plurality of learned models; as well as an arithmetic circuit capable of accessing the storage device, The multiple learned models include: The first model is learned in a manner to output an evaluation of the moving direction of the moving object with respect to input of one or more pieces of overall information based on the entire moving object and one or more pieces of first part information based on one or more first parts of the moving object in the image information; as well as The second model is learned in a manner that outputs an evaluation of the moving direction of the moving body relative to at least one of the one or more overall information, at least one of the one or more first part information, and one or more second part information based on the input of one or more second parts of the moving body, The one or more second portions are smaller than the one or more first portions, When the resolution of the image of the moving object detected from the object image information is less than a threshold value, the operation circuit selects the first model from the multiple learned models, and inputs the one or more overall information and the one or more first part information of the moving object detected from the object image information into the first model, so that the first model outputs an evaluation of the moving direction of the moving object detected from the object image information. When the resolution is above the threshold, the operation circuit selects the second model from the multiple learned models, and inputs at least one of the one or more overall information of the moving body detected from the object image information, at least one of the one or more first part information, and the one or more second part information into the second model, so that the second model outputs an evaluation of the moving direction of the moving body detected from the object image information.

Images