Information processing device, information processing method, and computer program

Abstract

To provide an information processing device that can reduce burdens required to set conditions for detecting anomalies.SOLUTION: An information processing device includes: object characteristic group information input means for inputting object characteristic group information including object characteristic information for at least two or more objects, the object characteristic information including type information representing a type of an object, position information of the object, and dwell time information indicating a time the object remains at the same position; an object arrangement characteristic database for retaining the type information, the position information, and the dwell time information in association with one other; and anomaly degree determination means for determining an anomaly degree based on the object characteristic group information and the object arrangement characteristic database, from the position information and the dwell time information included in the object characteristic group information.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a computer program. [Background technology]

[0002] Image recognition makes it possible to recognize object characteristics such as the type and position of an object from an image. In Patent Document 1, a bounding box surrounding an object is generated by image recognition, and the object type is output to update a database. Another example is an anomaly detection system that uses object recognition technology to detect, for example, a person entering a restricted area and output an alert to an administrator. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2019-517701 [Non-Patent Document 1] Jacob.et.al, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv 2018 [Non-patent document 2] Wald.et.al, Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions, CVPR2020 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional image recognition technology, when detecting an abnormality, it is necessary to set conditions for detecting the abnormality, which is time-consuming.

[0005] Therefore, an object of the present invention is to provide an information processing device that can save the effort of setting conditions for detecting an abnormality. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention includes: an object characteristic group information input means for inputting object characteristic group information including, for at least two or more objects, type information indicating the type of object, position information of the object, and staying time information for the object staying at the same position; an object placement characteristic database that stores the type information, the position information, and the staying time information in association with each other; an abnormality degree determining means for determining an abnormality degree from the position information and the staying time information included in the object characteristic group information based on the object characteristic group information and the object placement characteristic database; The present invention is characterized by having the following. [Effects of the Invention]

[0007] According to the present invention, it is possible to realize an information processing device that can eliminate the effort required for setting conditions for detecting an abnormality. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram showing a situation in which an information processing device according to a first embodiment of the present invention is used. [Figure 2] 1 is a functional block diagram showing the functional configuration of an information processing apparatus according to a first embodiment. [Figure 3] 1 is a diagram illustrating an example of the hardware configuration of an information processing apparatus 1 according to a first embodiment. [Figure 4] 4 is a flowchart showing the flow of an information processing method performed by the information processing apparatus of the first embodiment. [Figure 5] FIG. 2 is a schematic diagram showing the concept of calculating an abnormality degree in the first embodiment. [Figure 6] 10 is a flowchart showing an example of a processing procedure of step S403 by the abnormality degree determining unit 103 of the first embodiment. [Figure 7] FIG. 10 is a functional block diagram showing an example of the functional configuration of an information processing device according to Modification 1-3 of the first embodiment. [Figure 8] FIG. 10 is a functional block diagram showing the functional configuration of an information processing device 80 according to a second embodiment. [Figure 9] 10 is a flowchart showing the flow of an information processing method performed by the information processing device of the second embodiment. [Figure 10] FIG. 11 is a schematic diagram showing an example of a UI screen that is a processing result of the information processing apparatus according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same members or elements are designated by the same reference numerals, and duplicate descriptions will be omitted or simplified.

[0010] <Embodiment 1> Fig. 1 is a schematic diagram showing a situation in which an information processing device according to a first embodiment of the present invention is used. Fig. 1 shows an information processing device 1, a display unit 36, a user U, and an environment 10. The display unit 36 ​​displays, for example, the state of the environment 10 as a processing result of the information processing device 1.

[0011] A user U looks at the environment 10 displayed on the display unit 36 ​​to check the status of the object characteristic information, such as the degree of abnormality. The environment 10 shows a plan view of a store in a commercial facility as seen from above. The store includes multiple image capture devices C-1 and C-2, multiple shopping carts D-1 to Dk (k: an integer equal to or greater than 1), multiple product shelves T-1 to T-4, a floor F, and a wall W. Note that the number of image capture devices, product shelves, etc. is not limited to the example in FIG. 1.

[0012] The imaging devices C-1, C-2, etc. are, for example, network cameras, which capture images of the surrounding environment and distribute them over the Internet. The shopping carts D-1 to D-k are used by people (not shown) who push them. Hereinafter, the shopping carts will be simply referred to as carts.

[0013] Additionally, X-axis, Y-axis, and Z-axis are defined and displayed as coordinate axes relative to the floor F. In the first embodiment, attention is focused on a cart installed in a store, the degree of abnormality is calculated, and based on the calculated degree of abnormality, it is determined whether or not an abnormal state has occurred.

[0014] In this embodiment, abnormal conditions are detected based on the situation, such as whether the cart is installed in a specified position, whether it has been left behind, whether it is being used outside the usage area, etc. Object detection is realized by a neural network that generates a region of interest (a bounding box surrounding an object) based on image data and determines the type of object contained therein.

[0015] Conventional object detection methods have focused on detecting individual objects. This means that they are unable to utilize the relationships between objects (especially their positional relationships). In particular, there is common knowledge that "carts are used within the premises" and "unused carts are placed in designated locations," so "object placement characteristics" have been ignored when recognizing anomalies.

[0016] In contrast, in this embodiment, the "object placement characteristics" are calculated based on the positional relationships and positional information between objects in real space, the combination patterns and frequency of appearance of proximity relationships, the amount of time an object stays within a specified positional range, etc.

[0017] That is, in this embodiment, based on object placement characteristics that aggregate the "placement relationships of objects" in real space, a determination is made as to whether or not the state is normal based on information related to objects in real space and the relationships between those objects, and the degree of abnormality is determined based on the aggregated object placement characteristics.

[0018] Fig. 2 is a functional block diagram showing the functional configuration of the information processing device of embodiment 1. Note that some of the functional blocks shown in Fig. 2 are realized by causing a CPU or the like serving as a computer included in the information processing device to execute a computer program stored in a memory serving as a storage medium.

[0019] However, some or all of these functions may be implemented by hardware, which may be a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP).

[0020] 2 may not be contained in the same housing, but may be configured as separate devices connected to each other via signal paths. The above explanation regarding FIG. 2 also applies to FIG. 7 and FIG. 8.

[0021] 2, the information processing device 1 has an object characteristic group information input unit 101 as an object characteristic group information input means, an object placement characteristic database 102, and an abnormality degree determination unit 103 as an abnormality degree determination means. The object characteristic group information input unit 101 inputs object characteristic group information from, for example, a storage unit (not shown) that stores object characteristic group information, and outputs the input object characteristic group information to the abnormality degree determination unit 103.

[0022] The storage unit may be provided, for example, in a server external to the information processing device 1. The object characteristic information stored in the storage unit includes object type information in which a label is assigned to each type of object present in the environment (e.g., cart, wall, floor, etc.), three-dimensional position information in space consisting of the object's three-dimensional coordinates (X, Y, Z), and residence time information for the object to remain in the same position. Also, the object characteristic group information is information about multiple, at least two or more, objects present in a predetermined environment.

[0023] That is, the object characteristic group information input unit 101 inputs object characteristic group information containing object characteristic information for at least two or more objects, the object characteristic information including type information indicating the type of object, position information of the object, and residence time information for the object to remain within a predetermined position range.

[0024] The object placement characteristic database 102 is a database that stores placement characteristics that represent the positional relationships of multiple objects, and the placement characteristics refer to knowledge data that generalizes the three-dimensional positional relationships of objects in the real world. The object placement characteristic database 102 also stores type information that represents the type of object, position information of the object, and information on the duration of time that the object remains in the same position, in association with each other.

[0025] Specifically, placement characteristics such as "carts are used within the premises" and "unused carts are placed in specified locations" are stored in the object placement characteristic database 102. The placement characteristics stored in the object placement characteristic database 102 can also be said to be characteristics such as what kind of object placement is normal (usual) and what kind of placement is abnormal in reality, for example.

[0026] Furthermore, the object placement characteristic database 102 in this embodiment is a pre-trained neural network that is trained to estimate unknown object characteristics based on characteristic information of surrounding objects, or to infer normal object characteristic information.

[0027] Specifically, it is a neural network that stacks 24 layers of Ashish et al.'s Transformer ("Attention is All you Need", Ashish.et.el NeuralIPS2017).

[0028] In this embodiment, the number of input dimensions and the number of output dimensions of the Transformer are each 512. That is, a maximum of 512 pieces of object characteristic information can be input, and the same number of 512-dimensional output can be obtained.

[0029] Specifically, an encoder network used in the method of Jacob et al. (method described in Non-Patent Document 1) is used. (Note that the method of learning object placement characteristics of a neural network in this embodiment will be described later in embodiment 2.)

[0030] The abnormality degree determination unit 103 calculates and determines the degree of abnormality for each piece of object characteristic information included in the object characteristic group information, based on the object characteristic group information input to the object characteristic group information input unit 101 and the object placement characteristics held in the object placement characteristic database 102. That is, the abnormality degree determination unit 103 determines the degree of abnormality for at least one of the object type information, position information, and residence time information, based on the placement relationship with surrounding objects.

[0031] The calculation result of the degree of abnormality by the abnormality degree determination unit 103 is stored in a storage unit (not shown). The storage unit may be provided, for example, in a server or the like provided outside the information processing device 1. The higher the value of the degree of abnormality, the more abnormal the object characteristics of the target object and its surrounding objects.

[0032] That is, for example, if the cart has been parked for longer than a predetermined time, is located at a distance greater than a predetermined distance from the usage area, or both, this indicates an abnormal object characteristic. On the other hand, the lower the abnormality level, the closer the target object and its surrounding objects are to a normal location and normal park time.

[0033] Fig. 3 is a diagram showing an example of the hardware configuration of the information processing device 1 of embodiment 1. As shown in Fig. 3, the information processing device 1 has a CPU 31, a ROM 32, a RAM 33, an external memory 34, an input unit 35, a display unit 36, a communication I / F 37, an I / O 38, and a system bus 39 that interconnects the above devices. I / F is an abbreviation for interface. I / O is an abbreviation for Input / Output.

[0034] The ROM 32 stores a BIOS program and a boot program, and the RAM 33 is used as a temporary storage device for the CPU 31. The external memory 34 stores a computer program executed by the CPU 31, and the CPU 31 executes the computer program to control each device connected to the system bus 39 and perform the processing in this embodiment.

[0035] The input unit 35 performs processing related to input of information and the like from a keyboard, a mouse, etc. The keyboard, the mouse, etc. may be provided in the information processing device 1, or may be provided in an external device connected to the information processing device 1. The display unit 36 ​​outputs the calculation results of the information processing device 1, etc. to a display device in accordance with instructions from the CPU 31.

[0036] The display device may be a liquid crystal display device, a projector, an LED indicator, or the like, and may be provided in the information processing device 1 or may be provided externally to the information processing device 1.

[0037] The communication I / F 37 performs communication between the information processing device 1 and the outside. The object characteristic group information input unit 101 and the abnormality degree determination unit 103 input object characteristic group information and output abnormality degree determination results, etc., via the communication I / F 37.

[0038] The communication I / F 37 performs information communication via a network, but the type of communication interface is not important as long as it is Ethernet, USB, serial communication, wireless communication, etc. The I / O 38 performs input and output for each device connected to the system bus 39.

[0039] Fig. 4 is a flowchart showing the flow of an information processing method by the information processing device of embodiment 1. Note that the operation of each step in the flowchart of Fig. 4 is performed sequentially by a CPU or the like serving as a computer in the information processing device executing a computer program stored in memory.

[0040] The process of FIG. 4 is automatically started when the information processing device 1 is powered on and starts up.

[0041] In step S401, the information processing device 1 performs system initialization. That is, the CPU 31 reads and executes a program from the external memory 34, thereby putting the information processing device 1 into an operable state.

[0042] Furthermore, the CPU 31 reads weight parameters of the neural network, which is the object arrangement characteristic database 102, from the external memory 34 or the like as needed, and expands them in the RAM 33. After the series of initialization processes in step S401 are completed, the information processing device 1 executes the process of step S402.

[0043] In step S402 (object characteristic group information input step), the object characteristic group information input unit 101 inputs object characteristic group information from a storage unit provided in, for example, an external server, etc. The object characteristic group information input in step S402 includes object characteristic information for at least two or more objects, including type information indicating the type of object, position information of the object, and information on the duration of time the object remains in the same position.

[0044] Also, in step S402 , the object characteristic group information input unit 101 converts the input object characteristic group information into a data structure that can be recognized by the object arrangement characteristic database 102 , and outputs it to the abnormality degree determining unit 103 .

[0045] The data structure that can be recognized by the object placement characteristic database 102 refers to a data structure that has a feature vector (object type vector) that lists object type labels, a position vector that represents the position of those objects, and a dwell time vector that represents the time that the objects remain in the same position.

[0046] In step S403 (anomaly level determination step), the abnormality level determination unit 103 inputs the object type vector, the position vector, and the stay time vector into the object arrangement characteristic database 102, and executes an abnormality level determination process.

[0047] That is, in the abnormality degree determination step, the abnormality degree is determined from the position information and stay time information included in the object characteristic group information by inputting the object characteristic group information into the object placement characteristic database 102. Details of this abnormality degree determination process will be described later with reference to FIG.

[0048] In step S404, the information processing device 1 determines whether to end the flow of Fig. 4. If new object characteristic group information has been input, the information processing device 1 determines No in step S404, and if no new input has been made, the information processing device 1 determines Yes in step S404. If the information processing device 1 determines Yes in step S404, it ends the processing. If the information processing device 1 determines No in step S404, it returns to step S402.

[0049] Fig. 5 is a schematic diagram showing the concept of calculating the degree of abnormality in embodiment 1. Fig. 5 shows an overview of the process of inputting position information and residence time information for each object type into the input layer of the object placement characteristic database 102 and calculating the degree of abnormality.

[0050] Reference numeral 51 denotes object characteristic information. An object type vector 52, a position vector 53, and a dwell time vector 54 included in the object characteristic information 51 are input data for the object placement characteristic database 102. The object type vector 52 is an example of object type information.

[0051] The position vector 53 is an example of position information. The dwell time vector 54 is an example of dwell time information. The object type vector 52 input to the input layer of the object arrangement characteristic database 102 is composed of an object type information group consisting of multiple pieces of object type information.

[0052] In addition, the position information in this embodiment may include at least one of the position of an object in space, the relative position of objects in space, a label representing the position of an object in space, and a label representing the relative position of objects in space.

[0053] The object type vector 52 is a one-dimensional column vector. The first element of the object type vector 52 is a CLS token (a special label indicating the beginning of data) as shown in 50. Following the CLS token 50, the object type vector 52 includes an object label corresponding to each piece of object type information included in the object type information group.

[0054] 5, the object type vector 52 has, following the CLS token 50, an object label "Cart D-1" corresponding to the first object type information included in the object type information group. The object type vector 52 has, following the object label corresponding to the first object type information, an object label "Cart D-2" corresponding to the second object type information included in the object type information group.

[0055] The object type vector 52 stores the object type labels of all objects present in the environment in order from the last object type information included in the object type information group.

[0056] The position vector 53 is a one-dimensional column vector in which three elements, each representing the three-dimensional position X, Y, and Z of each object, are arranged. That is, each column of the position vector 53 stores the X, Y, and Z values ​​that are the position coordinates of the object in the corresponding column included in the object type vector 52.

[0057] The dwell time vector 54 sequentially stores values ​​indicating the time, for example, the number of seconds, that each object stays at the position coordinate of the position vector 53. The object placement characteristic database 102 inputs the object type vector 52, the position vector 53, and the dwell time vector 54, and obtains an output vector shown as 55.

[0058] The object placement characteristic database 102 outputs an output vector 55 predicted based on the weights of the neural network held in the object placement characteristic database 102. The output vector 55 includes an object type vector 56, a position vector 57, and a dwell time vector 58. In each column of the object type vector 56, an object type label predicted by the object placement characteristic database 102 is output.

[0059] First, the object characteristic information 51 input to the object placement characteristic database 102 is compared with the output vector 55 predicted by the object placement characteristic database 102. Then, if the object type of the input vector matches the object type of the output vector and the position of the input vector almost matches the position of the output vector, the degree of abnormality is calculated based on the difference between the dwell time of the input vector and the dwell time of the output vector.

[0060] That is, based on the object characteristic group information and the object placement characteristic database, the degree of abnormality is determined from the position information and the staying time information included in the object characteristic group information. More specifically, the degree of abnormality is determined based on the difference between the object characteristic group information input by the object characteristic group information input means and the object characteristic group information output by the object placement characteristic database.

[0061] Fig. 6 is a flowchart showing an example of the processing procedure of step S403 by the abnormality degree determination unit 103 in embodiment 1. Note that the operation of each step in the flowchart of Fig. 6 is performed sequentially by a CPU or the like serving as a computer in the information processing device executing a computer program stored in a memory.

[0062] In step S601, as described with reference to FIG. 5, object property group information consisting of a plurality of object property information is input to the input layer of the object arrangement property database 102.

[0063] In step S602, prediction is performed using the object layout characteristic database to obtain an output vector. That is, the calculation result is forward propagated using the neural network of the object layout characteristic database 102 to calculate the output vector.

[0064] In step S603, the object property group information input in step S601 is compared with the output vector calculated in step S602, column by column, excluding columns containing CLS tokens. That is, the comparison is performed as shown in the following equation 1.

[0065]

number

[0066] In step S604, the abnormality level of the residence time is calculated. Specifically, when ΔSm1=0 and ΔSm2 is less than a predetermined threshold, the abnormality level Es of each object is calculated using the following formula 2. Note that when ΔSm1≠0 or ΔSm2 is equal to or greater than the predetermined threshold, the abnormality level Es=0.

[0067]

number

[0068] The abnormality degree determining unit 103 calculates the abnormality degree for the object characteristic information input to the object arrangement characteristic database 102 through the processes of steps S601 to S604.

[0069] As described above, according to the first embodiment, it is possible to save the effort of setting conditions for detecting an abnormality.

[0070] <Variation 1-1> In the first embodiment, an example has been shown in which the abnormality degree of an object is determined based on the object characteristic group information input to the object characteristic group information input unit 101 and the object characteristic group information predicted by the object arrangement characteristic database 102.

[0071] However, there are cases where the input object type and the predicted object type differ when predicting object characteristic group information in the object placement characteristic database 102. In such cases, the degree of anomaly is determined as 1 if there is a difference between the input and output object types, and as 0 if there is no difference.

[0072] Furthermore, the object arrangement characteristic database 102 may also output a likelihood, which is the accuracy of the estimation. By using this likelihood in determining the degree of abnormality, the degree of abnormality for each object type can be expressed as a state other than whether or not there is an abnormality.

[0073] Specifically, the values ​​and likelihoods of each column of the object type vector 56, position vector 57, and dwell time vector 58 are stored. The degree of anomaly is calculated by multiplying the reciprocal of the likelihood by a predetermined coefficient. This makes it possible to calculate the degree of anomaly even when an anomaly appears in the object type, which is not numerical information.

[0074] <Variation 1-2> In the first embodiment, an example has been shown in which the abnormality degree of an object is determined based on the object characteristic group information input to the object characteristic group information input unit 101 and the object characteristic group information predicted by the object arrangement characteristic database 102.

[0075] However, in the prediction results of the object characteristic group information in the object placement characteristic database 102, there may be cases where the input position coordinates and the predicted position coordinates differ beyond a predetermined threshold range. In such cases, the degree of abnormality may be calculated based on the difference in position coordinates.

[0076] Specifically, similarly to the first embodiment, the object characteristic group information input to the object arrangement characteristic database 102 is compared with the object characteristic group information predicted by the object arrangement characteristic database 102. At this time, a difference value ΔP is calculated using the following formula 3 based on the input position coordinates (xm, ym, zm) and the predicted position coordinates (xom, yom, zom).

[0077]

number

[0078] Here, m=1, 2, ..., M (M is the number of objects, equal to the number of columns excluding CLS tokens). Then, the degree of abnormality Ep for the position coordinates may be calculated from the calculated ΔP using the following equation 4. Note that K2 is a predetermined fixed value.

[0079]

number

[0080] <Variation 1-3> In the first embodiment, object characteristic group information stored in a storage unit (not shown) is input. However, a configuration for generating object characteristic group information may be included. That is, the system may be configured to perform object recognition and include a configuration for generating an object type, position information of the object, and information on the duration of time the object remains in the same position.

[0081] Fig. 7 is a functional block diagram showing an example of the functional configuration of an information processing device according to Modification 1-3 of Embodiment 1. As shown in Fig. 7, in this modification, an image input unit 1001 as image input means for inputting an image, an object recognition unit 1002, and an object characteristic group information calculation unit 1003 are added to the configuration of Embodiment 1 shown in Fig. 2.

[0082] In this modification, image data groups are input at predetermined time intervals from the multiple image capture devices C shown in Fig. 1 to an image input unit 1001. The input image data groups are output to an object recognition unit 1002. The object recognition unit 1002 functions as an object recognition means that recognizes objects based on images, and assigns object labels to pixels by semantic segmentation based on the input image data groups, thereby recognizing the object type.

[0083] The object characteristic group information calculation unit 1003 extracts position information and residence time information as object characteristic group information for each object type recognized by the object recognition unit 1002. Specifically, the object characteristic group information calculation unit 1003 determines position coordinates in real space from the position information of the object type detected from the image data, and extracts them as object characteristic information.

[0084] Furthermore, the object characteristic group information calculation unit 1003 extracts staying time information from the position coordinates obtained from the input image data group. In this way, the object characteristic group information calculation unit 1003 calculates the extracted object type, the position information of the object, and the staying time information for the object to stay in the same position as object characteristic group information.

[0085] The object characteristic group information created as described above is input to the object characteristic group information input unit 101. Thereafter, the abnormality degree determination unit 103 determines and outputs an abnormality degree for each piece of object characteristic information included in the object characteristic group information, based on the object characteristic group information input to the object characteristic group information input unit 101 and the object arrangement characteristic database 102, as described in the first embodiment.

[0086] According to this modified example, by using the configuration shown in FIG. 7, it is possible to determine whether or not there is an abnormality in the surrounding situation by using the object placement characteristics to recognize the object type with high accuracy while outputting the degree of abnormality.

[0087] <Variation 1-4> In the first embodiment, the degree of abnormality of an object is determined based on the object characteristic group information input to the object characteristic group information input unit 101 and the object characteristic group information predicted by the object arrangement characteristic database 102. However, time information such as time period, day of the week, and date may also be used in predicting the object characteristic group information in the object arrangement characteristic database 102.

[0088] Specifically, time information is added to the object characteristic group information input to the object characteristic group information input unit 101. The object characteristic group information including the time information is input to the object arrangement characteristic database 102. In predicting the object characteristic group information, the object arrangement characteristic database 102 predicts the object characteristic group information taking into account differences in time zones from the input object characteristic group information and its time information. Thereafter, as described in the first embodiment, the abnormality degree determination unit 103 determines the abnormality degree of the object.

[0089] This allows for differences in common-sense object placement characteristics due to time information to be taken into consideration. For example, it is possible to detect abnormalities based on common-sense object placement patterns during and outside business hours of a store in a commercial facility. That is, for example, if a cart is not in its predetermined initial position outside business hours, it can be detected as an abnormality.

[0090] <Variation 1-5> In the first embodiment, the object placement characteristics are exemplified by the object type, the position coordinates in real space, and the duration of time the object stays at the position coordinates, but the present invention is not limited to this.

[0091] For example, the moving speed or the orientation of the object may be used. That is, for example, the information on the dwell time of the object placement characteristic described above may be replaced with the moving speed or the orientation of the object. Alternatively, characteristic information on the moving speed or the orientation of the object may be further added to the object placement characteristic described above.

[0092] For example, the speed at which an object moves can be used to determine the degree of abnormality by pre-accumulating the object movement speed in addition to the object placement characteristics described above in the object placement characteristic database 102. Specifically, in step S601, object characteristic group information including the object movement speed is input to the object placement characteristic database 102.

[0093] In step S602, object characteristic group information including the object moving speed is obtained from the object arrangement characteristic database 102. In step S603, the input object characteristic group information is compared with the output object characteristic group information obtained in step S602.

[0094] Then, just like with Equations 1 and 2, the difference in the object's moving speed is calculated, normalized by dividing it by the normal object speed, and the degree of anomaly of the normalized speed is calculated. This makes it possible to calculate the degree of anomaly regarding the object's moving speed, and objects moving at speeds different from the normal object speed can be detected without any extraction conditions.

[0095] Similarly, the orientation of an object can be used to determine the degree of abnormality by replacing or adding it to the object arrangement characteristics described above. Specifically, the orientation angle is stored in advance in the object arrangement characteristic database 102 as the orientation of the object.

[0096] The difference in the object's orientation is then found, and the degree of anomaly related to the object's orientation is calculated. Note that the posture angle, which is the object's orientation, does not change as an absolute value, so it becomes a normalized value using the difference in the object's orientation without dividing by the normal object orientation. This makes it possible to detect an anomaly in an object that is oriented differently from the normal object orientation (for example, upside down).

[0097] <Variation 1-6> In the first embodiment, the object placement characteristics are exemplified by the object type, the position coordinates in real space, and the duration that the object stays in the same position. However, it is also possible to add and input at least one of the size, weight, number, color, and temperature of the object as object characteristic information to the object placement characteristics database 102.

[0098] For example, height, width, depth, etc. may be input as characteristic information of the size of an object. Also, weight may be input as a characteristic of the weight of an object, quantity as a characteristic of the number of objects, color value as a characteristic of the color of an object, and temperature value as a characteristic of the temperature of an object.

[0099] Even when such object characteristic information is input, it is possible to calculate the degree of abnormality in the same manner as in Modification 1-2. This makes it possible to detect an abnormality when the input object characteristic information differs from the normal object characteristic information.

[0100] <Variation 1-7> In the first embodiment, in the processing of the abnormality degree determination unit 103, the abnormality degree is calculated based on the dwell time from the characteristic information included in the object characteristic group information input to the object placement characteristic database 102. However, the method of determining the abnormality degree does not need to be limited to this, and the abnormality degree may be determined based on a plurality of pieces of object characteristic information.

[0101] Specifically, three anomaly levels, namely, the anomaly level Ebm for the object type, the anomaly level Epm for the position coordinate, and the anomaly level Esm for the residence time, may be added together to calculate a single anomaly level Em, as shown in the following formula 5. This eliminates the need to set conditions for anomaly detection and makes it possible to detect anomalies when multiple object characteristic information items are combined.

[0102]

number

[0103] Here, m = 1, 2, ..., M (M is the number of objects, equal to the number of columns excluding CLS tokens). Furthermore, although Equation 5 is a simple summation, a weighting coefficient may be set for each item. Furthermore, as shown in Equation 6 below, the largest value of each item may be output as the degree of anomaly Em.

number

[0104] Furthermore, although the degree of abnormality for each object is determined in the first embodiment, the sum or average of the degrees of abnormality for each object may be calculated, thereby making it possible to determine the overall degree of abnormality for a scene including the group of objects input by the object characteristic group information input unit.

[0105] <Embodiment 2> In the first embodiment, a method for using the object arrangement characteristic database 102, which is a database of the arrangement of objects in real space, has been described. In the second embodiment, a method for generating (updating) the object arrangement characteristic database will be described.

[0106] 8 is a functional block diagram showing the functional configuration of an information processing device 80 of embodiment 2. In embodiment 2, in addition to the configuration described in embodiment 1, an object arrangement characteristic database update unit 81 that updates the object arrangement characteristic database 102, and a prediction unit 82 are added.

[0107] In this embodiment, the object characteristic group information input unit 101 inputs object characteristic group information and outputs it to the object arrangement characteristic database update unit 81. Unlike the first embodiment, the object characteristic group information input unit 101 outputs modified object characteristic group information, which is object characteristic group information obtained by modifying a part of the object characteristic information, to the prediction unit 82. The modification will be described later. The prediction unit 82 outputs the object characteristic group information predicted by the object arrangement characteristic database 102 to the object arrangement characteristic database update unit 81.

[0108] The object arrangement characteristic database update unit 81 updates the weights of the object arrangement characteristic database 102 based on the object characteristic group information input from the object characteristic group information input unit 101 and the object characteristic group information predicted by the prediction unit 82.

[0109] The object placement characteristic database update unit 81 outputs the updated weights to the object placement characteristic database 102. The object placement characteristic database update unit 81 functions as an object placement characteristic database generation unit that generates an object placement characteristic database.

[0110] Fig. 9 is a flowchart showing the flow of an information processing method by the information processing device of embodiment 2. Note that the operation of each step in the flowchart of Fig. 9 is performed sequentially by a CPU or the like serving as a computer in the information processing device executing a computer program stored in memory.

[0111] In the second embodiment, prediction processing in step S903 and object placement characteristic database update processing in step S904 are added to the processing procedures described in the first embodiment. The flow shown in Fig. 9 starts when an update start signal for the object placement characteristic database 102 is input by an input unit (not shown).

[0112] In step S901, which is the initialization process in this embodiment, the information processing device 80 performs initialization process of the object arrangement characteristic database 102 in addition to the initialization process in step S401 described in the first embodiment. That is, the information processing device 80 initializes the weights of the object arrangement characteristic database 102, which is a neural network. While any initialization method can be used, in this embodiment, initialization is performed with random values ​​generated from a normal distribution with a mean of 0 and a variance of 1.

[0113] In step S902, the object characteristic group information input unit 101 inputs object characteristic group information from, for example, a storage unit (not shown), as described in embodiment 1. Also in step S902, the object characteristic group information input unit 101 generates an object type vector, a position vector, and a residence time vector, as described in embodiment 1. In embodiment 2, some elements of these object type vector, position vector, and residence time vector are modified.

[0114] Specifically, the object characteristic group information input unit 101 selects one element from among the elements of the object type, position coordinates, and dwell time using a predetermined random number, and replaces it with a MASK token (a label indicating that the object type, position coordinates, or dwell time is unknown). The object characteristic group information input unit 101 outputs the object characteristic information modified in this way to the prediction unit 82. In addition, it outputs the object characteristic information before modification to the object arrangement characteristic database update unit 81.

[0115] In step S903, the prediction unit 82 performs a prediction process for the object characteristic information using the object placement characteristic database 102. That is, the object type vector, position vector, and residence time vector included in the modified object characteristic information are input to the object placement characteristic database 102. The object placement characteristic database 102 sequentially propagates the calculation results to the network, and obtains an output vector (predicted object placement characteristic).

[0116] In step S904, the object placement characteristic database update unit 81 updates the object placement characteristic database 102 based on the object placement characteristic predicted by the object placement characteristic database 102 and the object placement characteristic before modification.

[0117] In updating the object placement characteristic database 102, the weights of the neural network are learned using Diederik's method so that the difference between the object placement characteristic predicted in step S903 and the object placement characteristic before modification is reduced.

[0118] Diederik's method is Adam (Diederik et al., ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION, ICLR2015).

[0119] In step S905, the information processing device 80 determines whether or not to end the process, that is, whether or not to end the update of the object placement characteristic database 102. Specifically, the information processing device 80 executes the process of step S902 if the difference (prediction error) between the predicted object placement characteristic and the object placement characteristic before modification has decreased during the update process, and if not, ends the update flow in FIG.

[0120] As described above, in the second embodiment, the object placement characteristic database is updated so as to reduce the difference between the predicted object placement characteristic and the object placement characteristic before modification. By performing this processing, the object placement characteristic database 102 can predict unknown object characteristic information based on the placement characteristic of the object, i.e., the characteristic of the placement relationship with surrounding objects.

[0121] Furthermore, by using the updated object arrangement characteristic database 102, it is possible to grasp the normal surrounding situation with higher accuracy, and to obtain the degree of abnormality for the input object characteristic group information.

[0122] <Embodiment 3> In the first embodiment, a method for using an object arrangement characteristic database in which the arrangement of objects in real space is compiled into a database has been described. In the second embodiment, a method for generating (updating) an object arrangement characteristic database has been described. In the third embodiment, an example will be described in which the object arrangement characteristic database is used to visualize object characteristic group information and an abnormality degree when the abnormality degree is determined.

[0123] FIG. 10 is a schematic diagram showing an example of a UI screen that is the processing result of the information processing device of embodiment 3, and shows a UI (user interface) screen in which an abnormality degree is determined for the input object characteristic group information and the processing result is output to the display unit 36.

[0124] 10, the abnormality degree for each object's stay time is displayed as a data bar superimposed on the environment 10 of FIG. 1. Here, the abnormality degrees P101 to P105 for the stay time calculated for carts D-1 to D-5 are shown. The abnormality degrees P101 to P105 are shown as values ​​normalized to 0 to 100.

[0125] When the anomaly level value is low, it indicates that the object characteristic information is not abnormal, and the displayed bar is short. When the anomaly level value is high, it indicates that the object characteristic information is abnormal, and the displayed bar is long. In Figure 10, cart D-3 is shown to have a high value for the abnormality level P103 of the staying time, indicating that the staying time exceeds a reasonable value. In this way, the anomaly level of each object can be visualized.

[0126] In the third embodiment, the abnormality level of each object is visualized and displayed by the display unit 36, so that it is possible to easily grasp which object has an abnormality. Therefore, the user U can be alerted to the object in which an abnormality has occurred and its surroundings. Here, the display unit 36 ​​functions as a display means for presenting the abnormality level determined by the abnormality level determining means to the user.

[0127] In the above embodiment, the object characteristic information is shown as three-dimensional coordinates (X, Y, Z) for the position information of the object, but it may be two-dimensional coordinates instead of three-dimensional coordinates.

[0128] The present invention has been described above in detail based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0129] The present invention also includes those that realize the functions of the above-described embodiments using at least one processor or circuit such as a CPU, etc. Also, it is possible to use multiple processors to perform distributed processing.

[0130] In order to realize part or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an information processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the information processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0131] (Configuration 1) An information processing device characterized by comprising: an object characteristic group information input means for inputting object characteristic group information including, for at least two or more objects, object characteristic information including type information indicating the type of object, position information of the object, and residence time information for the object to remain in the same position; an object placement characteristic database that stores the type information, the position information, and the residence time information in association with each other; and an abnormality degree determination means that determines an abnormality degree from the position information and the residence time information included in the object characteristic group information based on the object characteristic group information and the object placement characteristic database.

[0132] (Configuration 2) The information processing device according to Configuration 1, wherein the position information includes at least one of the position of the object in space, the relative position of the object in space, a label representing the position of the object in space, and a label representing the relative position of the object in space.

[0133] (Configuration 3) The information processing device according to configuration 1 or 2, wherein the staying time information is information about the time during which the object stays within a predetermined position range.

[0134] (Configuration 4) An information processing device described in any one of configurations 1 to 3, characterized in that the abnormality degree determination means determines the abnormality degree of at least one of the type information, the position information, and the residence time information of the object based on its positional relationship with surrounding objects.

[0135] (Configuration 5) The information processing device described in any one of configurations 1 to 4, characterized in that the abnormality degree determination means determines the abnormality degree based on the difference between the object characteristic group information input by the object characteristic group information input means and the object characteristic group information output by the object placement characteristic database.

[0136] (Configuration 6) The information processing device according to any one of configurations 1 to 5, characterized in comprising: image input means for inputting an image; and object recognition means for recognizing the object based on the image.

[0137] (Configuration 7) The information processing device according to any one of configurations 1 to 6, further comprising an object placement characteristic database generating means for generating the object placement characteristic database.

[0138] (Configuration 8) The information processing device according to any one of configurations 1 to 7, further comprising a display means for presenting the degree of abnormality determined by the abnormality degree determining means to a user.

[0139] (Method) An information processing method characterized by comprising: an object characteristic group information input step of inputting object characteristic group information including, for at least two or more objects, object characteristic information including type information indicating the type of object, position information of the object, and residence time information during which the object remains in the same position; and an abnormality degree determination step of determining an abnormality degree from the position information and residence time information included in the object characteristic group information by inputting the object characteristic group information into an object placement characteristic database that stores the type information, the position information, and the residence time information in association with each other.

[0140] (Program) A computer program for controlling each of the means described in any one of configurations 1 to 8 by a computer. [Explanation of symbols]

[0141] 1: Information processing device 10:Environment 36: Display section C-1, C-2: Imaging device D-1~Dk: Cart U:User

Claims

1. an object characteristic group information input means for inputting object characteristic group information including, for at least two or more objects, type information indicating the type of object, position information of the object, and staying time information for the object staying at the same position; an object placement characteristic database that stores the type information, the position information, and the staying time information in association with each other; an abnormality degree determining means for determining an abnormality degree from the position information and the staying time information included in the object characteristic group information based on the object characteristic group information and the object placement characteristic database; An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the position information includes at least one of a position of the object in space, a relative position in space between the objects, a label representing the position of the object in space, and a label representing the relative position in space between the objects.

3. 2. The information processing apparatus according to claim 1, wherein the staying time information is information about a time during which the object stays within a predetermined position range.

4. 2. The information processing apparatus according to claim 1, wherein the abnormality level determining means determines the abnormality level of at least one of the type information, the position information, and the staying time information of the object based on a positional relationship with surrounding objects.

5. 2. The information processing apparatus according to claim 1, wherein the abnormality degree determining means determines the abnormality degree based on a difference between the object characteristic group information input by the object characteristic group information input means and the object characteristic group information output by the object placement characteristic database.

6. image input means for inputting an image; 2. The information processing apparatus according to claim 1, further comprising: an object recognition unit that recognizes the object based on the image.

7. 2. The information processing apparatus according to claim 1, further comprising an object placement characteristic database generating means for generating the object placement characteristic database.

8. 2. The information processing apparatus according to claim 1, further comprising display means for presenting the degree of abnormality determined by said abnormality degree determining means to a user.

9. an object characteristic group information input step of inputting object characteristic group information including, for at least two or more objects, object characteristic information including type information indicating the type of object, position information of the object, and residence time information for the object to stay at the same position; an abnormality degree determination step of determining an abnormality degree from the position information and the staying time information included in the object characteristic group information by inputting the object characteristic group information into an object placement characteristic database that stores the type information, the position information, and the staying time information in association with each other; An information processing method comprising:

10. A computer program for controlling each of the means according to any one of claims 1 to 8 by a computer.

Images