Computer system, method for controlling input of data to a model, and control device

Abstract

To realize control on data input to a model for obtaining a high-accuracy output.SOLUTION: A computer system uses object data indicating features of an object included in an image of surroundings of a control target to hold model information for use in management of a model determining a behavior of the control target and input control information for use in management of an input rule of the object data for the model based on a label given based on movement of the object, receives time-series data of the image, detects the object from the image, selects a target image, uses two or more images including the target image to calculate motion data indicating movement of the object detected from the target image, gives the label to the object detected from the image based on the motion data, and controls input of the object data for the model based on the input control information and the label.SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] The present invention relates to controlling the input of data into models that determine behavior. [Background technology]

[0002] Machine learning is being used to develop algorithms that realize automatic control of controlled objects such as vehicles, robots, and drones. For example, end-to-end deep reinforcement learning is being used to develop algorithms that output vehicle control signals from images.

[0003] End-to-end deep reinforcement learning has problems such as the need for a large amount of training data, the time required for training, and the fact that training is often difficult. Therefore, the development of algorithms using object-based reinforcement learning has attracted attention.

[0004] In object-based reinforcement learning, objects are extracted from an image, and a strategy for determining the behavior of the controlled object is learned based on the information about the extracted object.

[0005] In general, a policy is given as a model such as a neural network. When the model is a neural network, the neural network has multiple input slots. Usually, when the input slot for inputting object information changes, the output value also changes. In other words, this affects the accuracy of the policy. Therefore, it is important to control the input of object information to the input slots.

[0006] As a method for controlling the input of object information to an input slot, for example, the technique described in Non-Patent Document 1 is known.

[0007] Non-Patent Document 1 describes a method of determining an object score based on the object appearance information, selecting an object based on the score, and inputting the selected object information into a neural network. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] D. Wang, C. Devin, Q. -Z. Cai, F. Yu and T. Darrell, "Deep Object-Centric Policies for Autonomous Driving," 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8853-8859, doi: 10.1109 / ICRA.2019.8794224. Summary of the Invention [Problem to be solved by the invention]

[0009] Since the appearance of an object is highly diverse, it is necessary for a person to set the appearance of the object and the calculation method of the score, or to separately learn the calculation method of the priority. Therefore, there is a problem that the processing cost is high. In addition, the method described in Non-Patent Document 1 has a problem that the effect of improving the accuracy of the policy is small.

[0010] The present invention provides a computer system and method that allows for controlled input of object information into a model to obtain a highly accurate output. [Means for solving the problem]

[0011] A representative example of the invention disclosed in the present application is as follows: That is, a computer system includes at least one computer having an arithmetic unit, a storage device connected to the arithmetic unit, and an interface connected to the arithmetic unit, holds model information for managing a model that determines the behavior of a control target using object data representing features of objects included in an image surrounding the control target, and input control information for managing an input rule for the object data to the model based on a label assigned based on the movement of the object, receives time-series data of the images, detects the object from each of the images, selects a target image from among the images, calculates motion data representing the motion of the object detected from the target image using two or more images including the target image, stores the target image and the motion data in association with each other, assigns the label to the object detected from the image based on the motion data for each of the images, and controls input of the object data of the object detected from the image to the model based on the input control information and the label for each of the images. Effect of the Invention

[0012] According to the present invention, it is possible to realize input control of object data (object information) that improves the accuracy of model output. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0013] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer included in the computer system of the first embodiment. [Figure 2A] FIG. 2 is a diagram illustrating an example of a functional configuration of a computer according to a first embodiment. [Figure 2B] FIG. 2 is a diagram illustrating an example of a functional configuration of a computer according to a first embodiment. [Figure 3A] FIG. 2 is a diagram for explaining an example of an image handled by the computer system of the first embodiment. [Figure 3B]FIG. 2 is a diagram for explaining an example of an image handled by the computer system of the first embodiment. [Figure 3C] FIG. 2 is a diagram for explaining an example of an image handled by the computer system of the first embodiment. [Figure 4] 1 is a flowchart illustrating an example of a process executed by a computer according to the first embodiment. [Diagram 5] FIG. 13 is a diagram illustrating an example of a data structure of bounding box information generated by the computer according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a data structure of motion information generated by the computer of the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a data structure of object label information generated by the computer of the first embodiment. [Figure 8A] 4A to 4C are diagrams illustrating an example of input control of object data by an input control unit according to the first embodiment. [Figure 8B] 4A to 4C are diagrams illustrating an example of input control of object data by an input control unit according to the first embodiment. [Figure 9A] 4A to 4C are diagrams illustrating an example of input control of object data by an input control unit according to the first embodiment. [Figure 9B] 4A to 4C are diagrams illustrating an example of input control of object data by an input control unit according to the first embodiment. [Figure 10] 11 is a flowchart illustrating an example of a bounding box correction process executed by a bounding box prediction unit according to the first embodiment. [Figure 11] FIG. 4 is a diagram illustrating an example of a method for calculating a predicted bounding box by a bounding box prediction unit according to the first embodiment. [Figure 12A] FIG. 11 is a diagram illustrating an example of a functional configuration of a computer according to a second embodiment. [Figure 12B] FIG. 11 is a diagram illustrating an example of a functional configuration of a computer according to a second embodiment. [Figure 13] FIG. 11 is a diagram illustrating an example of a data structure of cluster information according to the second embodiment. [Figure 14] FIG. 11 is a diagram illustrating an example of a data structure of history information according to the second embodiment. [Figure 15] 13 is a diagram illustrating an example of input control of object data by an input control unit according to the second embodiment. FIG. [Figure 16] 13 is a flowchart illustrating a reclustering process executed by a computer according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention should not be interpreted as being limited to the description of the embodiment shown below. It will be easily understood by those skilled in the art that the specific configuration can be changed without departing from the concept or purpose of the present invention.

[0015] In the configurations of the invention described below, the same or similar configurations or functions are given the same reference symbols, and duplicated explanations are omitted.

[0016] In this specification, the terms "first," "second," "third," and the like are used to identify components and do not necessarily limit the number or order.

[0017] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings, etc. EXAMPLES

[0018] The computer system is configured from at least one computer 100. Fig. 1 is a diagram for explaining an example of the hardware configuration of the computer 100 included in the computer system of the first embodiment.

[0019] The computer 100 includes a processor 101, a memory 102, an auxiliary storage device 103, a network interface 104, and an input / output interface 105. The hardware elements are connected to each other via a bus.

[0020] The processor 101 is an arithmetic device that controls the entire computer 100, and executes programs stored in the memory 102. The processor 101 executes processes according to the programs, thereby operating as a functional unit (module) that realizes a specific function. In the following explanation, when a process is explained with a functional unit as the subject, it indicates that the processor 101 is executing a program that realizes the functional unit.

[0021] The memory 102 is a storage device that stores the programs executed by the processor 101 and information executed by the programs. The memory 102 is also used as a work area.

[0022] The auxiliary storage device 103 is a storage device that permanently stores data, and is, for example, a hard disk drive (HDD) and a solid state drive (SSD). The programs and information stored in the memory 102 may be stored in the auxiliary storage device 103. In this case, the processor 101 reads the programs and information from the auxiliary storage device 103 and loads them into the memory 102.

[0023] The network interface 104 is an interface for connecting to an external device via a network.

[0024] The input / output interface 105 is an interface for connecting to an input device 111 and an output device 112. Here, the input device 111 is a keyboard, a mouse, a touch panel, etc., and the output device 112 is a display, etc.

[0025] The computer 100 does not necessarily have to include the input / output interface 105 .

[0026] The computer system includes a computer 100 that learns a policy for determining an action, and a computer 100 that determines an action using the learned policy. Note that one computer 100 may perform both learning and action determination.

[0027] 2A and 2B are diagrams illustrating an example of a functional configuration of the computer 100 according to the first embodiment.

[0028] 2A shows a functional configuration of a computer 100-1 that learns a policy. The computer 100-1 has an object detection unit 200, a bounding box prediction unit 201, an object classification unit 202, a label assignment unit 203, an input control unit 204, and a learning unit 210. The computer 100-1 also holds classification definition information 230, occlusion detection definition information 231, label definition information 232, object data definition information 233, input control information 234, model information 235, and learning information 236. The computer 100-1 also receives self information 237 and reward information 238 from the outside. The self information 237 and reward information 238 are input together with the input of the image 250.

[0029] The classification definition information 230 is information for classifying objects. For example, the classification definition information 230 is definition information of a classifier. The occlusion detection definition information 231 is information on a method for detecting a state in which an object is occluded by another object. The label definition information 232 is information on a method for attaching a label based on the movement of an object. For example, the label definition information 232 is information for storing an entry in which an object type and a label attachment rule based on the movement of an object to be evaluated are associated with each other. The object data definition information 233 is information on the data structure of object data (feature amount) to be input when determining a policy. The input control information 234 is information on the input rule of object data for a model that realizes the policy. The model information 235 is definition information of the model. The learning information 236 is information for controlling the learning process of the policy. For example, the learning information 236 includes a learning rate and the like. The self information 237 is information on the internal state of the control target and the surrounding environment. The reward information 238 is information on the reward for the action.

[0030] The object detection unit 200 detects an object from an image 250 surrounding the control target. The object detection unit 200 generates a rectangular region (bounding box) that encompasses the object. The object detection unit 200 outputs the image 250 and the object's bounding box to a bounding box prediction unit 201 and an object classification unit 202, and outputs the image 250 to a label assignment unit 203.

[0031] The boundary box prediction unit 201 predicts a boundary box of an object using multiple images 250, compares the boundary box generated by the object detection unit 200 with the predicted boundary box, and corrects the boundary box based on the comparison result. The boundary box prediction unit 201 outputs the corrected boundary box to the label assignment unit 203.

[0032] The object classification unit 202 classifies the type of object included in the image 250. In this embodiment, the object classification unit 202 classifies the type of object for each bounding box. The object classification unit 202 outputs the classification result to the label assignment unit 203.

[0033] The label assignment unit 203 uses a plurality of images 250 to analyze the movement of an object included in the image 250, and assigns a label to the object included in the image 250 based on the movement of the object. The label assignment unit 203 outputs the object label and the image 250 to the input control unit 204.

[0034] The input control unit 204 generates object data 800 (see FIG. 8) for an object included in the image 250, and controls the input of the object data 800 to the model based on the label of the object.

[0035] The learning unit 210 learns a policy using the object data 800 input by the input control unit 204. The learning unit 210 includes a behavior decision unit 220 that decides an action based on the policy. The behavior decision unit 220 decides an action based on the policy defined by the model information 235, the self information 237, and the object data 800, and outputs the action as behavior data 260. The behavior data 260 is, for example, a control signal for controlling the torque and steering of the vehicle.

[0036] The learning unit 210 learns a policy based on the learning information 236, the reward information 238, and the action data 260, and reflects the learning result in the model information 235.

[0037] In the first embodiment, reinforcement learning is taken as an example for explanation, but a learning method other than reinforcement learning may also be used.

[0038] 2B shows the functional configuration of a computer 100-2 that determines an action using a policy. The computer 100-2 has an object detection unit 200, a bounding box prediction unit 201, an object classification unit 202, a label assignment unit 203, an input control unit 204, and an action determination unit 220. The computer 100-2 also holds classification definition information 230, occlusion detection definition information 231, label definition information 232, object data definition information 233, input control information 234, and model information 235, and also receives input of self information 237.

[0039] The functions and information possessed by the computer 100-2 are the same as those possessed by the computer 100-1, and therefore a description thereof will be omitted.

[0040] In addition, regarding each functional unit of the computers 100-1 and 100-2, multiple functional units may be integrated into one functional unit, or one functional unit may be divided into multiple functional units for each function. For example, the object detection unit 200 may include a bounding box prediction unit 201.

[0041] The computer system may learn a model and set the learned model to a control target such as a vehicle, a robot, or a drone. In this case, the control target includes a camera that captures an image, a driving device, and a control device such as a microcomputer having the functions shown in Fig. 2B. The control target controls the driving device based on the behavior data 260 output from the control device.

[0042] Next, an example of the image 250 handled by the computer system of the first embodiment will be described.

[0043] 3A, 3B, and 3C are diagrams for explaining an example of an image 250 handled by the computer system of the first embodiment.

[0044] 3A, the computer system of the first embodiment handles images captured by a camera attached to a vehicle 300. The camera captures images of a range 310. The range 310 includes objects (vehicles) 320-1, 320-2, 320-3, and 320-4. For the sake of explanation, a three-dimensional coordinate system is shown.

[0045] The camera outputs an image 250, for example, as shown in Figures 3B and 3C. Figure 3B shows the image 250 in a coordinate system whose origin is the upper left corner of the image. Figure 3C shows the image 250 in a coordinate system whose origin is the top center of the image.

[0046] The object detection unit 200 generates a bounding box 330 that encloses the detected object. From the image 250 in Figures 3B and 3C, bounding boxes 330-1, 330-2, 330-3, and 330-4 that enclose the objects 320-1, 320-2, 320-3, and 320-4 are generated.

[0047] In the image 250 of FIG. 3B and FIG. 3C, only vehicles are present as objects 320, but objects of different object types from vehicles, such as people, road signs, and traffic lights, may also be included.

[0048] Fig. 4 is a flowchart for explaining an example of a process executed by the computer 100-1 of the first embodiment. Fig. 5 is a diagram illustrating an example of a data structure of bounding box information generated by the computer 100-1 of the first embodiment. Fig. 6 is a diagram illustrating an example of a data structure of motion information generated by the computer 100-1 of the first embodiment. Fig. 7 is a diagram illustrating an example of a data structure of object label information generated by the computer 100-1 of the first embodiment.

[0049] The computer 100-1 acquires a plurality of images 250 (time-series data of the images 250) arranged in chronological order (step S101). The acquired images 250 are stored in a work area.

[0050] The object detection unit 200 of the computer 100-1 executes an object detection process for each image 250 (step S102).

[0051] In the object detection process, the object detection unit 200 detects one or more objects from one image 250 and generates a bounding box 330 for the detected objects. Note that a method for detecting an object from an image is a known technique, and therefore a detailed description thereof will be omitted.

[0052] The object detection unit 200 generates bounding box information 500 as a result of the object detection process for one image 250. The bounding box information 500 stores entries including an ID 501, a position 502, and a size 503. One entry corresponds to the bounding box 330 of one object. Note that the fields included in the entry are not limited to those described above. Any of the above-mentioned fields may not be included, and other fields may be included.

[0053] ID 501 is a field that stores identification information of the bounding box 330 (object). Position 502 is a field that stores the coordinate values ​​of the center of the bounding box 330. Note that instead of the coordinate values ​​of the center, the coordinate values ​​of the upper left corner of the bounding box 330 may be stored. Size 503 is a field that stores the width and height of the bounding box 330.

[0054] Furthermore, the object detection unit 200 refers to the bounding box information 500 of each image 250 and tracks the bounding boxes 330 to associate the bounding boxes 330 between the images 250. Based on the association, the object detection unit 200 assigns the same ID to the bounding boxes 330 of the same object.

[0055] The bounding box prediction unit 201 of the computer 100-1 executes a bounding box correction process (step S103) for each image 250. The bounding box correction process will be described with reference to FIGS.

[0056] The object classification unit 202 of the computer 100-1 executes an object classification process for each image 250 (step S104).

[0057] Since the classification of objects included in an image can be performed using a known technique, a detailed description thereof will be omitted. For example, a classification method using a Convolutional Neural Network (CNN) can be considered.

[0058] The labeling unit 203 of the computer 100-1 starts loop processing of the images 250 (step S105). Specifically, the labeling unit 203 selects one image 250 to be a target (target image 250). Here, it is assumed that the images 250 are selected in the past direction from the latest image 250. Note that the images 250 may be selected in the direction of time lapse.

[0059] The labeling unit 203 of the computer 100-1 starts a loop process of the object types (step S106). Specifically, the labeling unit 203 selects one object type to be a target (target object type).

[0060] The labeling unit 203 of the computer 100-1 calculates the motion data of the object of the target object type using the target image 250 (step S107). Specifically, the following process is executed.

[0061] (S107-1) The label assignment unit 203 reads out the image 250 that is immediately before the target image 250 in the time series.

[0062] (S107-2) The labeling unit 203 reads data on the bounding box 330 of the object of the target object type from the bounding box information 500 of each of the target image 250 and the read image 250.

[0063] (S107-3) The labeling unit 203 calculates an index representing the movement of the bounding box 330 of the object of the target object type, using the data of the bounding box 330. For example, the labeling unit 203 calculates the amount of change in the center of the bounding box 330 and the size (height and width) of the bounding box 330.

[0064] At this time, the labeling unit 203 may calculate a feature amount such as a collision margin time to be included in the object data 800. The collision margin time is calculated by any one of equations (1), (2), and (3).

[0065]

number

[0066]

number

[0067]

number

[0068] where w(t) represents the width of the bounding box 330 in the target image 250. Delta w represents the change in the width of the bounding box 330. h(t) represents the height of the bounding box 330 in the target image 250. Delta t represents the change in the height of the bounding box 330. s(t) represents the area of ​​the bounding box 330 in the target image 250. The area is given as the product of the width and the height. Delta s represents the change in the area of ​​the bounding box 330.

[0069] (S107-4) The label assignment unit 203 generates motion data composed of indices and registers it in the motion information 600.

[0070] 6, the motion information 600 stores entries including an ID 601, a position change amount 602, a size change amount 603, and a collision grace period 604. One entry corresponds to the motion data of one object. Note that the fields included in the entry are not limited to those described above. Any of the above-mentioned fields may not be included, and other fields may be included.

[0071] ID601 is the same field as ID501. Position change amount 602 is a field that stores the change amount of the center of bounding box 330. Size change amount 603 is a field that stores the change amount of the size of bounding box 330. Collision grace period 604 is a field that stores the collision grace period. As shown in Fig. 6, the motion data can be treated as a vector having indices as components.

[0072] It should be noted that the motion information 600 is generated for each object type. If the target image 250 is the oldest, there is no image 250 before that image 250, so the labeling unit 203 sets the index to 0. It should be noted that motion data may be calculated using three or more images 250 that are consecutive in time series.

[0073] This concludes the description of the process in step S107.

[0074] The label assignment unit 203 of the computer 100-1 assigns a label to the object of the target object type based on the motion information 600 and the label definition information 232, and generates object label information 700 (step S108).

[0075] In the label definition information 232, for example, the following rules for adding labels are defined. (Assignment rule 1) Labels are assigned from "L1" to "Ln" in descending order of the amount of change in the y-axis of the center. (Assignment rule 2) The motion data is sorted in order of the amount of change on the x-axis at the center, and labels from "L1" to "Ln" are assigned in order of the largest difference in the amount of change on the x-axis of the motion data at both ends. n represents the number of objects.

[0076] The labeling rule 1 is a method of labeling an object using the physical quantity contained in the motion data as an index. The labeling rule 2 is a method of labeling an object based on an index that indicates the difference from the motion of surrounding objects, which is calculated from the physical quantity contained in the motion data.

[0077] When the target image 250 is the oldest image, the label assignment unit 203 directly uses the label assignment result for the image 250 that is one image ahead in time series.

[0078] 7, the object label information 700 stores entries including an ID 701 and a label 702. One entry corresponds to one object. Note that the fields included in the entry are not limited to those described above. Any of the above-mentioned fields may not be included, and other fields may be included.

[0079] ID701 is the same field as ID501. Label 702 is a field for storing the determined label. Fig. 7 shows the result of assigning labels from "L1" to "Ln" in descending order of the amount of change on the y-axis of the center. n represents the number of objects.

[0080] The label assignment unit 203 of the computer 100-1 determines whether the process has been completed for all object types (step S109).

[0081] If it is determined that the process has not been completed for all object types, the label assignment unit 203 of the computer 100-1 returns to step S106 and executes the same process.

[0082] If it is determined that the processing has been completed for all object types, the label assignment unit 203 of the computer 100-1 determines whether the processing has been completed for all images 250 (step S110).

[0083] If it is determined that the processing has not been completed for all the images 250, the labeling unit 203 of the computer 100-1 returns to step S105 and executes the same processing.

[0084] When it is determined that the processing has been completed for all images 250, the input control unit 204 of the computer 100-1 generates object data 800 for the objects included in each image 250 based on the object data definition information 233 (step S111).

[0085] The input control unit 204 of the computer 100-1 controls the input of the object data 800 to the action determination unit 220 for each image 250 based on the object label information 700 and the input control information 234 (step S112).

[0086] Here, a description will be given of a method for controlling the input of object data to the action determining unit 220. Figures 8A, 8B, 9A, and 9B are diagrams for explaining an example of input control of object data 800 by the input control unit 204 in the first embodiment.

[0087] 8A and 8B, when the behavior decision unit 220 is a neural network composed of an input layer 810, a hidden layer 811, and an output layer 812, information indicating the association between a label and an input slot is stored in the input control information 234. In this case, the input control unit 204 inputs object data 800 to an input slot corresponding to the label of the object.

[0088] FIG. 8A shows input control of object data 800 when there is one object type, and FIG. 8B shows input control of object data 800 when there are two object types.

[0089] 9A and 9B, when the behavior decision unit 220 is composed of a recurrent neural network (RNN) 820 and a neural network (NN) 830, the input control information 234 stores information associating labels with the input order of object data 800 to the RNN 820. In this case, the input control unit 204 inputs the object data 800 to the RNN 820 in the order corresponding to the object labels.

[0090] FIG. 9A shows input control of object data 800 when there is one object type, and FIG. 9B shows input control of object data 800 when there are two object types.

[0091] The learning unit 210 of the computer 100-1 executes the learning process using the object data 800 whose input has been controlled (step S113). After that, the computer 100-1 ends the series of processes.

[0092] Since the learning process is a known technique, a detailed description will be omitted. The learning unit 210 may execute the learning process at a predetermined cycle rather than executing the learning process every time an image 250 is input. If the cycle has not elapsed, the processes of steps S112 and S113 can be omitted. If the learning process is not executed, the object data 800 is accumulated.

[0093] The process executed by the computer 100-2 is the same as that executed by the computer 100-1 in step S113, except that the process is replaced by an output process of the behavior data 260.

[0094] The labeling unit 203 assigns a label based on the movement of the object (boundary box 330). Since the movement of an object has less diversity than the appearance of the object, the meaning of the input can be clearly determined. For example, input rules for the object data 800 can be given, such as large horizontal movement, close to the host vehicle, fast moving speed, etc. By controlling the input of the object data 800 based on the input rules, it is expected that the learning efficiency and the output accuracy can be improved.

[0095] Fig. 10 is a flowchart illustrating an example of a boundary box correction process executed by the boundary box prediction unit 201 according to the first embodiment. Fig. 11 is a diagram illustrating an example of a method of calculating a predicted boundary box by the boundary box prediction unit 201 according to the first embodiment.

[0096] The boundary box prediction unit 201 selects a target image 250, and executes a boundary box correction process, which will be described below, on the target image 250. It is assumed that the image 250 is selected in the past direction from the latest image 250.

[0097] The bounding box prediction unit 201 starts loop processing of objects included in the target image 250 (step S201). Specifically, the bounding box prediction unit 201 selects one bounding box 330 from the bounding box information 500 of the target image 250.

[0098] The bounding box prediction unit 201 calculates a predicted bounding box 330 of the selected object using the target image 250 and the image 250 immediately preceding the target image 250 in the time series (step S202).

[0099] For example, the boundary box prediction unit 201 calculates a predicted boundary box 330 of the object 320-2 in the selected image 250 based on the boundary box 330 of the object 320-2 in the image 250(t1) that is chronologically preceding the target image 250(t2), as well as the size and moving speed of the object 320-1 that overlaps the object 320-2.

[0100] The predicted bounding box 330 can be calculated using known techniques, and a detailed description thereof will be omitted.

[0101] The bounding box prediction unit 201 calculates the error between the bounding box 330 of the selected object and the predicted bounding box 330 (step S203).

[0102] For example, the bounding box prediction unit 201 calculates the central x-axis error or the height error of the bounding box 330 .

[0103] The bounding box prediction unit 201 determines whether or not the selected object is occluded by another object based on the calculated error and the occlusion detection definition information 231 (step S204).

[0104] An error threshold is stored in the occlusion detection definition information 231, and the bounding box prediction unit 201 determines whether or not the selected object is occluded by another object based on a comparison result between the error and the threshold. For example, when the error is larger than the threshold, the bounding box prediction unit 201 determines that the selected object is occluded by another object.

[0105] If it is determined that the selected object is not occluded by another object, the bounding box prediction unit 201 proceeds to step S206.

[0106] If it is determined that the selected object is occluded by another object, the bounding box prediction unit 201 replaces the bounding box 330 of the selected object with the predicted bounding box 330 (step S205), and then proceeds to step S206.

[0107] Specifically, the bounding box prediction unit 201 sets the values ​​of the predicted bounding box 330 to the position 502 and size 503 of the entry selected from the bounding box information 500 .

[0108] By using the corrected bounding box 330 that takes into account the occlusion conditions, the object motion can be accurately estimated.

[0109] In step S206, it is determined whether or not the processing has been completed for all objects included in the target image 250 (step S206).

[0110] If it is determined that the processing has not been completed for all objects included in the target image 250, the bounding box prediction unit 201 returns to step S201 and executes the same processing.

[0111] If it is determined that the processing has been completed for all objects included in the target image 250, the boundary box prediction unit 201 ends the boundary box correction processing. EXAMPLES

[0112] The second embodiment differs from the first embodiment in that a label is assigned using a cluster configured from past object movement data. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0113] The hardware configuration of the computer 100 constituting the computer system of the second embodiment is the same as that of the first embodiment.

[0114] 12A and 12B are diagrams illustrating an example of a functional configuration of the computer 100 according to the second embodiment.

[0115] 12A shows the functional configuration of a computer 100-1 that learns a policy. The computer 100-1 of the second embodiment differs from the computer of the first embodiment in that it holds cluster information 239 and history information 240.

[0116] 12B shows the functional configuration of a computer 100-2 that determines an action using a policy. The computer 100-1 of the second embodiment differs from the computer of the first embodiment in that it holds cluster information 239 and history information 240.

[0117] FIG. 13 is a diagram showing an example of the data structure of the cluster information 239 according to the second embodiment.

[0118] Entries including a cluster ID 1301, a center of gravity 1302, and a range 1303 are stored. One entry corresponds to one cluster. Note that the fields included in the entry are not limited to those described above. Any of the fields described above may not be included, and other fields may be included.

[0119] The cluster ID 1301 is a field that stores identification information of a cluster. The center of gravity 1302 is a field that stores the center of gravity of the cluster. The range 1303 is a field that stores the size of the cluster.

[0120] In the second embodiment, a cluster in a feature space is assumed, the axes of which are the amounts of change in the x coordinate, y coordinate, width, and height included in the motion data. Note that the feature space is not limited to this.

[0121] A cluster may be formed for each object type. In this case, cluster information 239 exists for each object type.

[0122] FIG. 14 is a diagram illustrating an example of a data structure of the history information 240 according to the second embodiment.

[0123] The history information 240 stores entries including a position change amount 1401 and a size change amount 1402. One entry corresponds to motion data. Note that the fields included in an entry are not limited to those described above. Any of the above fields may not be included, and other fields may be included.

[0124] The position change amount 1401 and the size change amount 1402 are in the same field as the position change amount 602 and the size change amount 603 .

[0125] When clusters are formed for each object type, the entry includes a field for storing the object type.

[0126] In the label definition information 232 of the second embodiment, for example, the following assignment rules are defined. (Assignment rule 3) Assign the label “Ln” to cluster n. n represents the number of clusters.

[0127] The flow of processing executed by the computer 100-1 in the second embodiment is similar to that in the first embodiment, but there are some differences in the processing.

[0128] In step S 108 , the label assignment unit 203 assigns a label to the object of the target object type based on the motion information 600 , the label definition information 232 , and the cluster information 239 , and generates object label information 700 .

[0129] Specifically, the label assignment unit 203 determines a cluster to which the object belongs based on the object motion data and the cluster information 239. The label assignment unit 203 stores the motion data in the history information 240. The label assignment unit 203 assigns a label to an object of the target object type based on the cluster and label definition information 232.

[0130] The input of the object data 800 in the second embodiment can be controlled by the same control method as in the first embodiment. Also, a control method as shown in FIG.

[0131] FIG. 15 is a diagram illustrating an example of input control of object data 800 by the input control unit 204 in the second embodiment.

[0132] The behavior determining unit 220 is composed of an RNN 820 and an NN 830. In this case, the label definition information 232 defines a label assignment rule for each cluster. Also, the input control information 234 stores information that associates the label of each cluster with the input order of the object data 800 to the RNN 820. In this case, the input control unit 204 inputs the object data 800 of each cluster to the RNN 820 in the order corresponding to the object label.

[0133] FIG. 16 is a flowchart illustrating the reclustering process executed by the computer 100-1 of the second embodiment.

[0134] The computer 100-1 executes the reclustering process when a predetermined number of motion data are accumulated, when a predetermined time has elapsed, or when an execution request is received. Here, it is assumed that the labeling unit 203 executes the reclustering process. However, it may be executed by another functional unit.

[0135] The label assignment unit 203 executes clustering using the history information 240 (step S301). A known method may be used for the clustering, and therefore a detailed description thereof will be omitted.

[0136] The label assignment unit 203 refers to the cluster information 239 and determines whether or not the previous cluster can be associated with the new cluster (step S302).

[0137] For example, if the error in the center of gravity between the previous cluster and the new cluster is smaller than a threshold value for all clusters and the error in the range is smaller than a threshold value, the label assignment unit 203 determines that the previous cluster can be associated with the new cluster.

[0138] If it is determined that the previous cluster can be associated with the new cluster, the label assignment unit 203 updates the cluster information 239 (step S303) and ends the reclustering process.

[0139] Specifically, the label assignment unit 203 updates the center of gravity 1302 and range 1303 of each cluster. Since the clusters are substantially unchanged before and after reclustering, the label definition information 232 and the input control information 234 do not need to be changed.

[0140] If it is determined that the previous cluster cannot be associated with the new cluster, the label assignment unit 203 updates the cluster information 239 (step S304).

[0141] Specifically, the label assignment unit 203 initializes the cluster information 239, and sets an entry of a new cluster in the cluster information 239 based on the result of clustering.

[0142] The label assignment unit 203 requests the user to reset the label definition information 232 and the input control information 234 (step S305).

[0143] In the second embodiment, since a label is set according to the cluster to which the label belongs, when the cluster is changed, it is necessary to reset the label definition information 232 and the input control information 234. Therefore, when the cluster is changed, the label assignment unit 203 prompts the user to reset the label definition information 232 and the input control information 234.

[0144] The label assignment unit 203 receives the input of the label definition information 232 and the input control information 234 from the user (step S306). After that, the label assignment unit 203 ends the reclustering process.

[0145] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment describes the configuration in detail to easily explain the present invention, and the present invention is not necessarily limited to the configuration including all the described configurations. Also, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0146] In addition, each of the above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0147] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Python, Java (registered trademark), etc.

[0148] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.

[0149] In the above-mentioned embodiment, the control lines and information lines are shown as those considered necessary for the explanation, and not all the control lines and information lines are shown in the product. All the components may be connected to each other. [Explanation of symbols]

[0150] 100 calculator 101 Processor 102 Memory 103 Auxiliary storage device 104 Network Interface 105 Input / Output Interface 111 Input Device 112 Output Device 200 Object detection unit 201 Bounding Box Prediction Unit 202 Object classification section 203 Labeling Department 204 Input control section 210 Learning Department 220 Action Decision-Making Department 230 Classification definition information 231 Occlusion detection definition information 232 Label definition information 233 Object Data Definition Information 234 Input Control Information 235 Model Information 236 Learning Information 237 Self-information 238 Reward Information 239 Cluster Information 240 History Information 250 images 260 Behavioral Data 300 vehicles 310 Range 320 Object 330 Bounding Box 500 Bounding box information 600 Movement Information 700 object label information 800 object data 810 Input Layer 811 Hidden Layer 812 Output layer 820 Recurrent Neural Networks 830 Neural Networks

Claims

1. 1. A computer system comprising: At least one computer having a computing device, a storage device connected to the computing device, and an interface connected to the computing device; The system holds model information for managing a model that determines a behavior of a control target using object data that represents features of objects included in an image of the control target, and input control information for managing input rules for the object data to the model based on labels that are assigned based on the movement of the object, receiving time series data of the image; Detecting the object in each of a plurality of the images; selecting a target image from the plurality of images, calculating motion data representing a motion of the object detected from the target image using two or more of the images including the target image, and storing the target image and the motion data in association with each other; for each of a plurality of said images, applying said label to said object detected in said image based on said motion data; A computer system comprising: a computer that controls, for each of a plurality of said images, input of said object data of said object detected from said image to said model based on said input control information and said label.

2. 2. The computer system of claim 1, The motion data is a vector having physical quantities representing the motion of the object as components, The computer system comprises: holding label definition information for managing a rule for assigning the label based on an index calculated using the motion data; Calculating the indicator using the motion data of the object detected from the image; A computer system comprising: a computer that assigns the label to the object detected from the image based on the index and the label definition information.

3. 2. The computer system of claim 1, The motion data is a vector having physical quantities representing the motion of the object as components, The computer system comprises: holding cluster information for managing clusters configured from a plurality of pieces of motion data in a feature space having an axis of the physical quantity included in the motion data, and label definition information for managing a rule for assigning the label based on the cluster to which the cluster belongs; Identifying the cluster to which the object detected from the image belongs based on the motion data and the cluster information; A computer system comprising: a computer that assigns the label to the object detected from the image based on the identified cluster and the label definition information.

4. 2. The computer system of claim 1, A bounding box is calculated as an area including the object included in the image; using two or more of the images including the target image, selecting a target object from among the objects included in the target image, and determining whether the target object is occluded by another of the objects; correcting the bounding box of the target object if the target object is occluded by another of the objects; A computer system comprising: a computer that calculates the motion data of the object detected from the target image based on an amount of change in the bounding box of the object in two or more images including the target image.

5. 2. The computer system of claim 1, Accepting time series data of learning images for learning the model; Detecting the object from each of the plurality of learning images; selecting a target learning image from the plurality of learning images, calculating the motion data of the object detected from the target learning image using two or more learning images including the target learning image, and storing the target learning image and the motion data in association with each other; for each of the plurality of training images, assigning the label to the object detected from the training image based on the motion data; for each of the plurality of learning images, controlling input of the object data of the object detected from the learning image to the model based on the input control information and the label; A computer system comprising: a learning process that uses the object data whose input is controlled, thereby updating the model; and a result of the update that is reflected in the model information.

6. A method for controlling input of data to a model, the method being executed by a computer system, comprising: The computer system comprises: At least one computer having a computing device, a storage device connected to the computing device, and an interface connected to the computing device; The system holds model information for managing a model that determines a behavior of a control target using object data that represents features of objects included in an image of the control target, and input control information for managing input rules for the object data to the model based on labels that are assigned based on the movement of the object, The data input control method includes: a first step of receiving, by the at least one computer, time series data of the image; a second step of the at least one computer detecting the object in each of a plurality of the images; a third step in which the at least one computer selects a target image from the plurality of images, calculates motion data representing a motion of the object detected from the target image using two or more of the images including the target image, and stores the target image and the motion data in association with each other; a fourth step of the at least one computer assigning, for each of a plurality of the images, the labels to the objects detected in the images based on the motion data; a fifth step of controlling, for each of a plurality of the images, input of the object data of the object detected from the image to the model based on the input control information and the label by the at least one computer; 13. A method for controlling input of data to a model, comprising:

7. A method for controlling input of data to a model according to claim 6, comprising the steps of: The motion data is a vector having physical quantities representing the motion of the object as components, the computer system holds label definition information that manages a rule for assigning the label based on an index calculated using the motion data; The fourth step includes: said at least one computer calculating said indicia using said motion data of said object detected from said image; and a step in which the at least one computer assigns the label to the object detected from the image based on the index and the label definition information.

8. A method for controlling input of data to a model according to claim 6, comprising the steps of: The motion data is a vector having physical quantities representing the motion of the object as components, the computer system holds cluster information for managing clusters configured of a plurality of pieces of motion data in a feature space having an axis of the physical quantity included in the motion data, and label definition information for managing a rule for assigning the label based on the cluster to which the cluster belongs; The fourth step includes: identifying, by the at least one computer, the cluster to which the object detected from the image belongs based on the motion data and the cluster information; assigning the labels to the objects detected in the image based on the identified clusters and the label definition information, by the at least one computer; 13. A method for controlling input of data to a model, comprising:

9. A method for controlling input of data to a model according to claim 6, comprising the steps of: The second step includes: said at least one computer calculating a bounding box that is an area that contains said object in said image; the at least one computer using two or more of the images including the target image to select a target object from among the objects included in the target image and determine whether the target object is occluded by another of the objects; and correcting the bounding box of the target object if the target object is occluded by another of the objects, A method for controlling input of data to a model, characterized in that the third step includes a step in which the at least one computer calculates the motion data of the object detected from the target image based on the amount of change in the bounding box of the object in two or more images including the target image.

10. A method for controlling input of data to a model according to claim 6, comprising the steps of: The at least one computer receives time-series data of training images for training the model; detecting the object from each of a plurality of training images by the at least one computer; a step of the at least one computer selecting a target learning image from the plurality of learning images, calculating the motion data of the object detected from the target learning image using two or more learning images including the target learning image, and storing the target learning image and the motion data in association with each other; the at least one computer assigning the label to the object detected from the training image based on the motion data for each of the training images; a step of controlling, for each of the plurality of learning images, an input of the object data of the object detected from the learning image to the model based on the input control information and the label, by the at least one computer; updating the model by executing a learning process using the object data whose input is controlled by the at least one computer, and reflecting a result of the update in the model information; 13. A method for controlling input of data to a model, comprising:

11. A control device that determines an action of a control target, A computing device, a storage device connected to the computing device, and an interface connected to the computing device, The method includes: storing model information for managing a model that determines a behavior of the control target using object data that represents features of objects included in an image of the control target; and storing input control information for managing input rules for the object data to the model based on labels that are assigned based on the movement of the object; receiving time series data of the image; Detecting the object in each of a plurality of the images; selecting a target image from the plurality of images, calculating motion data representing a motion of the object detected from the target image using two or more of the images including the target image, and storing the target image and the motion data in association with each other; for each of a plurality of said images, applying said label to said object detected in said image based on said motion data; A control device characterized by controlling, for each of a plurality of images, input of the object data of the object detected from the image to the model based on the input control information and the label.

12. The control device according to claim 11, The motion data is a vector having physical quantities representing the motion of the object as components, The control device includes: holding label definition information for managing a rule for assigning the label based on an index calculated using the motion data; Calculating the indicator using the motion data of the object detected from the image; A control device which assigns the label to the object detected from the image based on the index and the label definition information.

13. The control device according to claim 11, The motion data is a vector having physical quantities representing the motion of the object as components, The control device includes: holding cluster information for managing clusters configured from a plurality of pieces of motion data in a feature space having an axis of the physical quantity included in the motion data, and label definition information for managing a rule for assigning the label based on the cluster to which the cluster belongs; Identifying the cluster to which the object detected from the image belongs based on the motion data and the cluster information; A control device which assigns the label to the object detected from the image based on the identified cluster and the label definition information.

14. The control device according to claim 11, A bounding box is calculated as an area including the object included in the image; using two or more of the images including the target image, selecting a target object from among the objects included in the target image, and determining whether the target object is occluded by another of the objects; correcting the bounding box of the target object if the target object is occluded by another of the objects; A control device comprising: a control unit that calculates the motion data of the object detected from the target image based on an amount of change in the bounding box of the object in two or more images including the target image.

Images