Information processing method, analysis device, imaging system, and analysis system

Abstract

This information processing method executes, by means of a computer: a process for acquiring a captured image; a process for setting detection frames with respect to a plurality of objects, including a vehicle, which are at least some of the objects appearing in the captured image, the detection frames indicating coordinate ranges in which the plurality of objects are present, respectively; a process for specifying an attribute of an object indicated by each detection frame; and a process for converting an attribute of the vehicle or of another object in accordance with the attribute of the vehicle or the other object if the detection frame of the vehicle and the detection frame of the other object are superimposed.

Description

Information processing method, analysis device, imaging system, and analysis system

[0001] The present disclosure relates to an information processing method, an analysis device, an imaging system, and an analysis system.

[0002] There are technologies for detecting objects in images. For example, Patent Literature 1 discloses a video surveillance system that uses a classifier that has learned the characteristics of people carrying belongings or the belongings to determine whether a person in an image captured by a surveillance camera is carrying a specific belonging, determines different attributes of the person depending on the belongings, and outputs the results of statistical processing based on the attribute determination results.

[0003] Patent No. 7047944

[0004] In one aspect, an object is to provide an information processing method and the like that can suitably perform object detection.

[0005] In one aspect, the information processing method includes a computer executing the following processes: a process of acquiring an image; a process of setting a detection frame indicating the coordinate range in which an object, which is at least a portion of the object shown in the image and includes a vehicle, is located; a process of identifying the attributes of the object indicated by each of the detection frames; and a process of converting the attributes of the vehicle or other object according to the attributes of the other object or vehicle when the detection frame of the vehicle overlaps with the detection frame of another object.

[0006] In one aspect, object detection can be suitably performed.

[0007] FIG. 1 is a diagram illustrating an example of the configuration of an analysis system. FIG. 2 is a block diagram illustrating an example of the configuration of an analysis device. FIG. 3 is a diagram illustrating an overview of an embodiment. FIG. 4 is a diagram illustrating an example of attribute conversion processing of a detection frame. FIG. 5 is a diagram illustrating an example of attribute conversion processing of a detection frame. FIG. 6 is a diagram illustrating an object tracking process. FIG. 7 is a diagram illustrating a process of counting the number of passes. FIG. 8 is a flowchart illustrating an example of a processing procedure executed by the analysis device.

[0008] 1 is a diagram showing an example of the configuration of an analysis system. In this embodiment, an analysis system is described that detects objects (e.g., passersby) from an image captured by a camera (captured image) and tracks the movement of the objects to count the number of objects (e.g., the number of people) that pass through a specific line or area. The analysis system includes an imaging system 1 and a server 2 (management device). The imaging system 1 and the server 2 are communicatively connected via a network N such as the Internet.

[0009] The imaging system 1 includes an analysis device 10 and a camera 20. The camera 20 is an imaging device that captures images of passersby and the like, and is installed, for example, on a street or inside a building. The analysis device 10 performs object detection and object tracking of objects included in the image captured by the camera 20, and counts the number of objects that pass through a certain line or area based on the results of the detection and tracking. In this embodiment, as an example, the analysis device 10 counts the number of people who pass through a count line (virtual line) that is virtually set in the area captured by the camera 20 (i.e., the number of people passing through the line).

[0010] The server 2 is a server computer that acquires the analysis results of the images obtained by the analysis device 10 (in this embodiment, the number of passersby) and manages the analysis results. For example, the analysis device 10 counts the number of passersby at predetermined intervals and transmits the analysis results to the server 2. At this time, the analysis device 10 may send the images captured by the camera 20 to the server 2, or may discard them without sending them. In this embodiment, the analysis device 10 sends only the analysis results to the server 2 and discards the captured images. In response to a request from a user, the server 2 outputs the analysis results and / or analysis results obtained by further analyzing the analysis results to a display device such as a terminal device (not shown).

[0011] FIG. 2 is a block diagram showing an example configuration of the analysis device 10. The analysis device 10 includes a control unit 11, a main memory unit 12, a communication unit 13, an input / output unit 14, and an auxiliary memory unit 15. The control unit 11 has one or more processors, such as a field programmable gate array (FPGA), a central processing unit (CPU), or a graphics processing unit (GPU), and performs various information processing by reading and executing a program P stored in the auxiliary memory unit 15. The main memory unit 12 is a temporary storage area, such as a random access memory (RAM), and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing and transmits and receives information to an external device (e.g., a server 2). The input / output unit 14 is an input / output interface connecting the analysis device 10 and the camera 20. The input / output unit 14 acquires captured images from the camera 20 and outputs control signals to the camera 20 for controlling the camera 20. The camera 20 and the analysis device 10 may be connected via a wired or wireless connection.

[0012] When acquiring a captured image from camera 20, analysis device 10 may also acquire time information and / or an identifier of camera 20. This time information and / or identifier may be output from camera 20, or may be identified on the analysis device 10 side. Then, analysis device 10 may output the analysis result and the time information and / or the identifier to server 2 in association with each other.

[0013] The auxiliary storage unit 15 is a non-volatile storage area such as a ROM (Read Only Memory), and stores a program P (a program product) and other data for the control unit 11 to execute processing. The auxiliary storage unit 15 also stores a detection module 51 and an identification module 52.

[0014] The detection module 51 is a program module used for object detection, and includes one or more machine-learned models (object detection models) that are trained to detect each object in an input image. More specifically, when an image is input, the detection module 51 identifies a detection frame (bounding box) indicating the coordinate range in which each object in the image may exist, and the object attributes (e.g., the class to which the object is predicted to belong) indicated (by each detection frame), and outputs them in association with each other.

[0015] The output format of the detection frame and attributes is not particularly limited. For example, if the detection frame is rectangular, the position and size of the detection frame may be output as the coordinates of its four corner vertices, or as the coordinates of a vertex and the length and width from that vertex. Furthermore, when identifying the attributes of an object, the detection module 51 may select an attribute that is predicted to be most likely to be the attribute of the object from a predetermined group of attributes and identify it as the attribute of the object.

[0016] The detection module 51 may be, for example, a YOLO (You Only Look Once) program module. The trained model may be a type of CNN (Convolutional Neural Network). The object detection model may be a neural network other than a CNN, or a machine learning model with an architecture other than a neural network. The object detection method is not limited to a machine learning model, and object detection (i.e., setting a detection frame for a captured image and identifying the attributes of an object indicated by the detection frame) may be achieved using, for example, rule-based pattern matching.

[0017] The object detection model may also be composed of multiple trained models. For example, multiple types of object detection models specialized in detecting specific objects may each be configured to set a detection frame and identify attributes, and the results may be combined to provide the object detection result. For example, an object detection model specialized in detecting people as "objects" and an object detection model specialized in detecting automobiles may each be used to set a detection frame and identify attributes, and the combined results may be used as the object detection result.

[0018] The identification module 52 is a program module including one or more trained machine learning models (identification models) used for object tracking. A discrimination model is a model that has been trained to extract image features (image features) when an image is input. The discrimination model of the identification module 52 can be implemented, for example, by DeepSORT. In this embodiment, a machine learning model (identification model) is used to extract image features of people (objects). However, the model may be a model other than DeepSORT, and extraction methods other than machine learning models may also be used. The target from which features are extracted may be objects other than people, such as vehicles, or both people and objects other than people. The discrimination model may also be composed of multiple trained models. For example, multiple types of discrimination models specialized in extracting image features of specific objects may each extract image features, and the combined results may be used in a downstream process for object tracking. For example, a discrimination model specialized in extracting image features of people and a discrimination model specialized in extracting image features of automobiles may each extract image features from the same image, and the combined results may be used in a downstream process for object tracking.

[0019] The analysis device 10 is not limited to the above configuration and may include, for example, an input unit that accepts operation input and / or a display unit that displays images. The analysis device 10 may also include a reading unit that reads the portable storage medium 1a and reads and executes the program P from the portable storage medium 1a. In this case, part or all of the detection module 51 and / or the identification module 52 may be stored in the portable storage medium 1a rather than in the auxiliary storage unit 15.

[0020] 3 is a diagram showing an outline of the embodiment, and the outline of the embodiment will be described with reference to FIG.

[0021] The analysis device 10 sequentially acquires video captured on the street from the camera 20. For example, the analysis device 10 analyzes one minute of video (i.e., one minute of time-series frame images) at predetermined intervals (e.g., every minute), counts the number of people who pass through the virtual line, and transmits (outputs) the number of people and / or analysis results including the number of people to the server 2 as needed. The analysis device 10 may count the number of people who pass through the count line by attribute of the people who pass through, or may count the total number of people regardless of attribute. The analysis device 10 may also count both the total number of people who pass through and the number of people who pass by attribute and transmit the counts to the server 2.

[0022] First, the analysis device 10 detects objects shown in each frame image of a video captured by the camera 20. In this embodiment, the analysis device 10 performs object detection using a detection module 51. When an image is input, the detection module 51 sets a detection frame (so-called a bounding box) for each object in the image and further identifies the attributes of the object that are indicated (or predicted to be indicated) by the detection frame. The detection module 51 then outputs these in association with each other. In other words, the detection module 51 assigns a detection frame to each object in the image, and the detection frame is assigned the attributes of the object indicated by the detection frame. Hereinafter, the "attributes of the object indicated by the detection frame" may also be referred to as the "attributes of the detection frame" or the "attributes assigned to the detection frame."

[0023] The analysis device 10 sequentially inputs frame images to the detection module 51, causing the detection module 51 to set a detection frame for each object in each frame image and identify the attributes of each detection frame. For example, if the detected object is a person, an attribute such as "Person" is identified; if the detected object is a bicycle, an attribute such as "Bicycle" is identified; and if the detected object is a car, an attribute such as "Car" is identified. The attribute "Person" is an example of an attribute indicating a person. In addition, attributes indicating a person may include more specific attributes such as passerby or store clerk, or attributes that further subdivide age and gender such as male, female, adult, child, and elderly. All of these are examples of "attributes indicating a person." The attributes "Bicycle" and "Car" are also examples of attributes indicating a vehicle. In addition, attributes indicating a vehicle may include more specific attributes that subdivide vehicles such as motorcycle, tricycle, light car, and truck. All of these are also examples of "attributes indicating a vehicle."

[0024] Next, the analysis device 10 performs a process to convert the attributes of the detected object. More specifically, the analysis device 10 first determines whether the detection frame of the vehicle and the detection frame of another object overlap. If it determines that they overlap, the analysis device 10 converts the attributes of at least one of the overlapping detection frames to attributes different from the original attributes. During this attribute conversion, the analysis device 10 converts the attributes of the other detection frames according to (the content of) at least one attribute of each of the overlapping detection frames. In this embodiment, the attributes of the other detection frames are converted according to the attributes of the detection frame of the vehicle.

[0025] Furthermore, when it is determined that three or more detection frames overlap, if at least one of the detection frames satisfies the second condition, the attribute conversion described above may be performed. Furthermore, regarding the second condition, for example, when two vehicle detection frames overlap, the attribute conversion described above may or may not be performed. For example, when a detection frame of a car overlaps with a detection frame of another car, the analysis device 10 may determine that each is a separate car detection frame without performing attribute conversion (and merging or deleting the detection frames, as described below).

[0026] The specific attributes of the vehicle that are converted into the attributes of other objects can be determined as appropriate. For example, these can be determined as appropriate depending on the specifications of the detection module 51 (e.g., the types of objects that can be detected and / or that are the target of attribute assignment) and how the final object detection results are used.

[0027] In this embodiment, the detection module 51 is a model that can detect one or more types of vehicles and people as "objects." In this embodiment, an example will be described in which, when a vehicle detection frame and a person detection frame overlap, the attributes of the person detection frame are converted according to the attributes of the vehicle detection frame. However, the objects that the detection module 51 can detect are not limited to this. It is sufficient for the detection module 51 to be able to detect at least some of the objects that appear in the captured image.

[0028] 4A to 4C are diagrams schematically illustrating an example of the attribute conversion process for detection frames. In each of the diagrams, rectangles represent detection frames, and words adjacent to the detection frames represent the attributes of each detection frame. As shown in FIGS. 4A to 4C, when a vehicle detection frame and a person detection frame overlap, the analysis device 10 according to this embodiment converts the attribute representing the person according to the attribute of the vehicle on which the detection frame overlaps.

[0029] In the example of Fig. 4A, when a detection frame for an object whose attribute is "Person" overlaps with a detection frame for an object whose attribute is "Bicycle," the attribute of "Person" is converted to an attribute of "Bicycle" based on the attribute of "Bicycle." In the example of Fig. 4B, when a detection frame for an object whose attribute is "Person" overlaps with a detection frame for an object whose attribute is "Motorcycle," the attribute of the person detection frame is converted to an attribute of "Motorcycle" based on the attribute of "Motorcycle."

[0030] On the other hand, in the example of Figure 4C, two detection frames of objects with the attribute "person" (Person) are superimposed on the detection frame of an object with the attribute "car." In this case, the analysis device 10 may perform attribute conversion determination and processing separately for the detection frames of the two other objects (persons) superimposed on the car detection frame, or may process them together. In either case, as in Figures 4A and 4B, the attribute "person" is converted to the attribute "car occupant" (Occupant) based on the attribute "car."

[0031] When performing this attribute conversion process, the analysis device 10 may determine the converted attributes based on the attributes of the vehicle detection frame (i.e., the vehicle attributes) and the attributes of the object that overlaps with the vehicle detection frame (i.e., the attributes of the other object). For example, if the detection module 51 is a model that can detect one or more types of vehicles and multiple types of objects including people, the attributes of the person detection frame may be converted on the condition that the vehicle detection frame and the person detection frame overlap. In this case, conversion similar to the attribute conversion shown in Figures 4A to 4C can be performed.

[0032] Furthermore, when detecting objects other than humans, such as animals such as dogs and cats, and / or structures such as utility poles and signs, the above-described attribute conversion may not be performed if a vehicle detection frame overlaps with a non-human detection frame. For example, if a human detection frame overlaps with a bicycle detection frame, the attribute of the human detection frame may be converted to "Bicycler," while if an animal detection frame overlaps with a bicycle detection frame, the attribute of neither detection frame may be converted. Also, attribute conversion may be performed when a human detection frame does not overlap with a vehicle detection frame. For example, if a human detection frame does not overlap with a vehicle detection frame, the analysis device 10 may convert the attribute "Person" to the attribute "Walker."

[0033] The analysis device 10 deletes the detection frame of the object whose attribute is not converted (the detection frame of the vehicle in the examples of FIGS. 4A to 4C ) at any timing before or after the attribute conversion. As a result, in the examples of FIGS. 4A to 4C , only the detection frame of the person is extracted and remains, and this remaining detection frame of the person can be tracked during object tracking, as described below. Therefore, when it is desired to count the number of people by object tracking, for example, it is expected that the accuracy of object tracking can be improved and / or the load can be reduced. In this way, according to this embodiment, by leaving only detection frames suitable for subsequent processing, it is possible to perform subsequent processing in an optimal manner.

[0034] 3, the analysis device 10 performs processing to identify the movement trajectory of each person by tracking the position of the detection frame of the person (object) detected in each frame image between frames. In this embodiment, the analysis device 10 performs object tracking using a Person ReID (Person Re-Identification) technique.

[0035] Fig. 5 is a diagram showing the object tracking process, which extracts image features of a person detected from a frame image, calculates the degree of match (similarity) between the image features of the person and image features of a person detected in a frame prior to the current frame (e.g., the frame immediately preceding the current frame, in which object tracking was performed), and assigns the same ID (identifier) ​​to the same person according to the degree of match.

[0036] For example, analysis device 10 processes a plurality of time-series frame images captured by camera 20 in the order in which they were captured. First, analysis device 10 extracts image features of each person from the image area within the detection frame of each person detected in the first frame image. In this embodiment, analysis device 10 extracts image features of each person by inputting images of the image area within the detection frame of each person to identification module 52.

[0037] Because the current frame is the first frame image (i.e., the first frame image in the video to be subjected to object detection), analysis device 10 assigns a new ID indicating each person to each detection frame in the current frame. Analysis device 10 stores information such as image feature amounts, coordinates, and attributes of each person in a database in association with the ID.

[0038] When processing the next frame, analysis device 10 extracts image features of each person by inputting the image of the image area within the detection frame of each person detected in the current frame to identification module 52. Analysis device 10 then compares the image features of each person in the current frame with the image features of each person detected in the frame before the current frame, and assigns the same ID to the same person.

[0039] Specifically, the analysis device 10 calculates the similarity (e.g., cosine distance) between the image feature of each person in the current frame and the image feature of each person in the frame preceding the current frame. Note that the "similarity" is not limited to the cosine distance, and for example, cosine similarity may be used. If the calculated similarity is within a threshold, the analysis device 10 determines that the person appearing in the current frame is the same person as the person appearing in the previous frame and assigns the same ID. On the other hand, if the similarity is equal to or greater than the threshold, the analysis device 10 determines that the person appearing in the current frame is a person not appearing in the previous frame and assigns a new ID. The analysis device 10 stores information such as the person's image feature, detection frame coordinates, and attributes in a database in association with the ID.

[0040] The analysis device 10 performs the above process for each frame and assigns an ID identifying each person to the detection frame of each person in each frame. The analysis device 10 then identifies the movement trajectory of the person by tracking the position of the detection frame assigned the same ID between frames.

[0041] The analysis device 10 may improve the accuracy of object tracking by considering not only the degree of coincidence (similarity) of image features within the detection frame, but also the predicted movement position of the detection frame.

[0042] Specifically, the analysis device 10 uses a Kalman filter to estimate the movement position of the detection frame in the next frame (current frame) following the multiple frames, based on the positions of detection frames assigned the same ID in the multiple frames prior to the current frame. The analysis device 10 calculates the distance (e.g., Euclidean distance) between the position of the detection frame of the person detected in the current frame and the movement position of the detection frame estimated from the multiple frames prior to the current frame. The analysis device 10 then assigns an ID indicating a person to each detection frame based on the calculated distance and the similarity of the image features.

[0043] Specifically, the analysis device 10 determines whether the calculated distance is within a threshold and whether the similarity is within a threshold. If it is determined that the distance is within the threshold and the similarity is within the threshold, the analysis device 10 determines that the person detected in the current frame is the same person as the person detected in the previous frame, and assigns the same ID to the detection frame. On the other hand, if it is determined that the similarity is within the threshold but the distance is equal to or greater than the threshold, the analysis device 10 determines that the person detected in the current frame is a different person from the person detected in the previous frame, and assigns a new ID to the detection frame. The analysis device 10 identifies the movement trajectory of the person by tracking the position of the detection frame assigned the same ID between frames.

[0044] In this way, the analysis device 10 may perform object tracking by taking into consideration not only the degree of match of the image feature amounts but also the degree of match of the detection frame positions (distance from the predicted movement position).

[0045] Continuing the explanation, returning to Fig. 3, the analysis device 10 then performs processing to count the number of passing people for each attribute and / or the number of passing people (traffic volume) for all people from the movement trajectories of people with each attribute.

[0046] 6 is a diagram showing the process of counting the number of passes, illustrating the movement trajectory of a person and a virtual line (count line) set in the image.

[0047] The virtual line is a line virtually set within the imaging range of the camera 20 and can be arbitrarily set by the user. The virtual line is not limited to a straight line and may be a curved line. Furthermore, the virtual line may not be a two-dimensional line but may be an area (i.e., a virtual area) with an area. The analysis device 10 counts the number of passing people for each attribute based on whether the trajectory of the person with each attribute intersects with the virtual line. Specifically, as shown in FIG. 6 , the analysis device 10 identifies the person's movement trajectory from the position of the detection frame at each time point (frame) and determines whether the person's trajectory intersects with the virtual line for each time point. For example, the analysis device 10 divides the image into "Area 1" and "Area 2" using the virtual line and counts the number of people entering "Area 2" from "Area 1" and the number of people exiting "Area 1" from "Area 2" (see FIG. 1 ). The counting may be performed by attribute or the total for all attributes. That is, the analysis device 10 counts the total number of objects (here, people) that pass through the virtual line and / or the number for each attribute.

[0048] The analysis device 10 transmits the analysis results, including the total count and / or the number for each attribute, to the server 2. At this time, the analysis device 10 may output the analysis results by associating them with time information and / or the identifier of the camera 20. In this embodiment, the analysis device 10 discards the captured images. The server 2 outputs the analysis results in response to a request from the user. For example, the server 2 may display the analysis results on a display device or the like.

[0049] Fig. 7 is a flowchart showing an example of a processing procedure executed by the analysis device 10. The processing content executed by the analysis device 10 will be described with reference to Fig. 7. For convenience, Fig. 7 shows a series of processing steps from start to finish for a video acquired from the camera 20 at one time, but this series of processing steps is repeatedly executed as long as captured images are continuously supplied from the camera 20.

[0050] The camera 20 outputs captured images (e.g., videos) to the analysis device 10. The camera 20 may send captured images to the analysis device 10 at any timing of its own device, or may output captured images at a timing instructed by the control unit 11 of the analysis device 10. The captured images sent from the camera 20 to the analysis device 10 may be accumulated in chronological order in a storage device. The control unit 11 of the analysis device 10 temporarily stores the captured images in an internal or external memory, and the control unit 11 can read and use these in chronological order as appropriate in subsequent processing. Note that if the processing speed of the analysis device 10 is sufficient, the control unit 11 may receive the captured images output from the camera 20 as needed and perform processing from step S12 onwards, without saving them in a storage device or the like.

[0051] The control unit 11 of the analysis device 10 acquires a captured image (step S11). As described above, "acquisition" here may refer to direct acquisition from the camera 20 or to reading out a captured image that has been stored in a storage device or the like. The control unit 11 acquires one or more frame images.

[0052] The control unit 11 selects one of the multiple frame images in the order in which each frame is captured (step S12). Note that if the control unit 11 itself loads the captured images frame by frame, the loaded frame image is used as is for subsequent processing, and the processing of step S12 may be omitted. The control unit 11 inputs the selected frame image to the detection module 51 to detect each object in the frame image (step S13). Specifically, the control unit 11 assigns a detection frame (bounding box) to each object in the frame image, which is a detection frame that indicates the range in which the object exists and indicates the attributes of the object.

[0053] When the detection frame of a vehicle overlaps with the detection frame of a person (another object) among the detected objects, the control unit 11 converts the attribute of the person according to the attribute of the vehicle (step S14). The control unit 11 assigns the converted attribute to the person detection frame and deletes the vehicle detection frame from the detection result (step S15).

[0054] The control unit 11 extracts image features of the person (object) that is the subject by extracting an image area within the detection frame from the frame image and inputting the extracted image area to the identification module 52 (step S16). The control unit 11 calculates the similarity between the image features of each person detected in the current frame and the image features of each person detected in the frame before the current frame (step S17). Note that if the current frame is the first frame, the control unit 11 skips the processing of step S17.

[0055] The control unit 11 assigns an ID (identifier) ​​indicating the detected person to each detection frame in the current frame according to the calculated similarity (step S18). Specifically, if the similarity is equal to or less than a threshold, the control unit 11 assigns the same ID as the person detected in the previous frame to the detection frame. If the similarity is equal to or greater than the threshold, the control unit 11 assigns a new ID to the detection frame. If the current frame is the first frame, the control unit 11 assigns a new ID to the detection frame. The control unit 11 stores information such as the image features of the subject person, the coordinates and attributes of the detection frame, etc., in association with the ID in the database (step S19). If the control unit 11 has acquired multiple frames (i.e., frames for a predetermined period) in step S11, the control unit 11 repeatedly executes the processes of steps S12 to S19 for the multiple frames. On the other hand, if the control unit 11 has acquired only one frame in step S11, the control unit 11 repeats the processes of steps S12 to S19 for all frames for the predetermined period. Note that the predetermined period is set as appropriate and is a period equivalent to at least two frames.

[0056] The control unit 11 determines whether the processes of steps S13 to S19 have been performed for all frames for a predetermined period (step S20). If it determines that the processes have not been performed for all frames (S20: NO), the control unit 11 returns to step S12. In this case, the control unit 11 selects the next frame image (step S12) and performs the processes of steps S13 to S19.

[0057] If it is determined that processing has been performed on all frames (S20: YES), the control unit 11 identifies the movement trajectory of each person by tracking the position of the detection frame assigned the same ID between frame images (step S21). The control unit 11 counts the number of people passing by each attribute by determining whether the trajectory of the person with each attribute intersects with a virtual line in the image (step S22). The control unit 11 outputs (transmits) the analysis result including the counted number of people passing to the server 2 (step S23), and ends the series of processes for the captured image acquired in step S11.

[0058] After the process of S23, or at any timing parallel to steps S11 to S23, the analysis device 10 acquires the next captured image to be processed from the camera 20 (step S11). Then, the process of steps S12 to S23 is continuously executed for this newly acquired captured image. That is, for example, if the camera 20 is capturing video in real time and outputting it to the analysis device 10, the process shown in FIG. 7 loops back to step S11 after the process of step S23 (or step S12 if a captured image has been acquired earlier), and is repeated for the new captured image.

[0059] On the other hand, if the camera 20 intermittently captures video or still images and sends them to the analysis device 10, the analysis device 10 may perform processing from step S11 onwards when it acquires an image from the camera 20 (or stores the image in a memory device), and once the processing is completed, it may wait until the next image is input from the camera 20 (or stored in a memory device).

[0060] Note that analysis device 10 may perform the processes of steps S13 to S19 on all frame images of the acquired video, or may perform the processes of steps S13 to S19 on frames extracted at any interval from all frames, thereby determining that all frames have been processed. For example, if a 10-second video with a frame rate of 5 fps is acquired in step S11, analysis device 10 may perform the processes of steps S13 to S19 a total of 10 times, every 5 frames, to determine that all frames of this 10-second video have been processed in step S20.

[0061] 7, the processing from step S16 onward does not require the direct use of the frame image selected in step S12. Therefore, the control unit 11 may discard the frame image that has undergone the processing of steps S13 to S16 as appropriate. Furthermore, the processing of steps S16 to S19 for a certain frame image (referred to as the first frame image) and the processing of steps S13 to S15 for the frame image selected as the next processing target (referred to as the second frame image) may be performed in parallel. In other words, while processing related to object tracking is being performed on the first frame image, processing related to object detection may be performed on the second frame image.

[0062] Furthermore, if the captured image output at one time is a plurality of frame images, the control unit 11 may divide the plurality of frame images into predetermined periods and perform the processes of steps S12 to S19 for the video for that predetermined period. Then, once these processes are completed, the control unit 11 may determine in S20 that the processes have been performed for all frames for that predetermined period. For example, if the video captured by the camera 20 is five minutes long, the control unit 11 may determine YES in step S20 after completing the processes of steps S12 to S19 for one minute of the video, and then perform the processes of steps S21 to S23. This process may be repeated five times.

[0063] As described above, according to this embodiment, the detection frame of a vehicle and the detection frame of a person (other object) riding in the vehicle are integrated, and an attribute corresponding to the vehicle is assigned to the person. This makes it possible to appropriately determine the attributes of the object indicated by the detection frame during object detection. Furthermore, by integrating or deleting one or more of the superimposed detection frames, it is possible to extract a detection frame suitable for subsequent processing (e.g., object tracking and headcount, measurement of the number of people and / or vehicles passing by, etc.). In other words, it can be said that object detection can be performed suitably. Furthermore, after object detection, object tracking, headcount, measurement of the number of people passing by, etc. can be performed suitably.

[0064] (Modification) In the above-described embodiment, an example was described in which, when a vehicle detection frame and a person detection frame overlap, the attributes of the person are converted. However, when a vehicle detection frame and a person detection frame overlap, the analysis device 10 according to this embodiment may convert the attributes of the object indicated by the vehicle detection frame, retain the vehicle detection frame, and delete the person detection frame. Using the example of FIG. 4A as an example, for example, the analysis device 10 may convert the attribute of the "bicycle" detection frame to "bicycle with person on board" and delete the "person" detection frame. In this way, it is sufficient for the analysis device 10 to convert the attributes of the person or vehicle according to the attributes of the vehicle or person. In this case, as in the above-described example, a detection frame suitable for object detection can be extracted, thereby enabling suitable object detection. Furthermore, after object detection, object tracking, counting people, measuring traffic volume, and the like can be performed suitably.

[0065] Furthermore, after attribute conversion, the analysis device 10 may integrate multiple overlapping detection frames and assign the converted attributes to the integrated detection frames, rather than deleting detection frames with attributes that were not converted. For example, the analysis device 10 may set only the overlapping portion of the vehicle detection frame and the person detection frame as a new detection frame, or may set a new detection frame that includes the vehicle detection frame and the person detection frame. When detection frames are integrated in this way, the integrated detection frame is no longer a frame that indicates the presence area of ​​a single object, and therefore the converted attributes may be assigned to the integrated detection frame. As such, various methods are conceivable for integrating detection frames.

[0066] Furthermore, when deleting (or merging) detection frames and / or converting attributes, the analysis device 10 may perform the merging (or deletion) of detection frames and / or attribute conversion by referring to the object detection results in previous (preferably, the most recent one or more) frame images. For example, if attribute conversion is performed by referring only to the determination results of a single frame, detection frames may be merged even when a pedestrian happens to pass in front of a bicycle, and the pedestrian's attribute may be converted to "bicycle rider." Therefore, the analysis device 10 may prevent such unintended attribute conversion and deletion (or merging) of detection frames by referring to whether vehicle detection frames and person detection frames overlap in other frames (preferably, frames processed immediately before and / or after steps S13 to S19). For example, if the detection frame of an object assigned the attribute "bicycle" moves between frames, the analysis device 10 converts the object's attribute to "motorcycle rider."

[0067] Specifically, when the analysis device 10 determines whether a vehicle detection frame and a person detection frame overlap in each frame, it stores the position or identifier of the person detection frame and the determination result. When the analysis device 10 determines that a vehicle detection frame and a person detection frame overlap in one frame, it reads the determination results for the other frames. The analysis device 10 then determines whether a detection frame having the same position or identifier as the person detection frame overlaps with the vehicle detection frame in a predetermined number or more (e.g., more than half) of frames. When it determines that there is overlap in a predetermined number or more frames, the analysis device 10 converts the person attributes according to the vehicle attributes and deletes the vehicle detection frame. In contrast, when it determines that there is no overlap in a predetermined number or more frames, the analysis device 10 does not convert the attributes and does not delete the vehicle detection frame. This prevents unintended deletion or merging of detection frames.

[0068] Furthermore, when determining whether or not the detection frames overlap, the analysis device 10 may calculate an index value of the degree of overlap between the detection frame of the vehicle and the detection frame of another object, and refer to this index value to determine whether or not to perform attribute conversion. For example, the analysis device 10 may convert the attribute when the index value is a percentage equal to or greater than a threshold, and may not convert the attribute when the index value is less than the threshold.

[0069] An example of an index value for the degree of overlap is Intersection over Union (IoU). In this case, the analysis device 10 may convert the attributes of detection frames whose IoU with the vehicle detection frame is equal to or greater than a predetermined threshold, and may not convert the attributes of detection frames whose IoU is less than the threshold even if they overlap.

[0070] Another example of the index value of the degree of overlap may be the "proportion of the area where the detection frame of the vehicle overlaps with the detection frame of another object to the entire area of ​​the detection frame of the vehicle." Here, the index value of this proportion is referred to as "IoV (Intersection over Vehicle)." When IoV is used as the index value, it is also possible to determine whether or not to perform attribute conversion using a processing procedure similar to that for IoU described above.

[0071] Another example of an index value for the degree of overlap may be the "proportion of the area overlapping with the vehicle detection frame to the entire area of ​​the person detection frame." Here, the index value for this proportion is referred to as "IoP (Intersection over Person)." When IoP is employed, the analysis device 10 identifies whether or not the detection frames of other objects overlapping with the vehicle detection frame are human detection frames during attribute conversion. In other words, the analysis device 10 refers to the attributes of the other objects overlapping with the vehicle detection frame. The subsequent processing is the same as the IoU and IoV described above, even when IoP is employed as the index value.

[0072] When using such an index value of the degree of overlap for judgment, only the attribute of the object indicated by the detection frame with the largest index value may be converted. For example, suppose three person detection frames overlap a single vehicle detection frame. In this case, the analysis device 10 converts only the attribute of the one detection frame with the largest index value of the degree of overlap (e.g., IoU, IoV, or IoP) among the three person detection frames, and does not need to convert the attributes of the remaining two person detection frames. Note that if the index values ​​for two or more detection frames are the same, the attributes of all detection frames with the same value may be converted. Specifically, in the example of FIG. 4C , the index values ​​for the two "Person" detection frames are the same, so the attribute of both detection frames may be converted to "Occupant."

[0073] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0074] The features described in each embodiment can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the form of reference.

[0075] REFERENCE SIGNS LIST 1 Imaging system 10 Analysis device 11 Control unit 12 Main memory unit 13 Communication unit 14 Input / output unit 15 Auxiliary memory unit P Program 51 Detection module 52 Identification module 20 Camera 2 Server (management device)

Claims

1. An information processing method in which a computer executes the following processes: a process of acquiring a captured image; a process of setting detection frames indicating the coordinate ranges in which at least some of the objects appearing in the captured image, including vehicles, exist; a process of identifying the attributes of the objects indicated by each of the detection frames; and a process of converting the attributes of the vehicle or other object in accordance with the attributes of the other object or vehicle when the detection frame of the vehicle and the detection frame of another object overlap.

2. The information processing method according to claim 1, wherein the conversion process converts attributes of the other object in accordance with attributes of the vehicle.

3. The information processing method according to claim 2, wherein the conversion process converts the attributes of the other object in accordance with the attributes of the vehicle and the attributes of the other object.

4. The information processing method according to claim 1, wherein the conversion process converts the attributes of the vehicle in accordance with the attributes of the other object.

5. An information processing method according to any one of claims 2 to 4, wherein the conversion process converts attributes when an index value of the degree of overlap between the detection frame of the vehicle and the detection frame of the other object is equal to or greater than a threshold, and does not convert attributes when the index value is less than the threshold.

6. The information processing method according to claim 5, wherein the conversion process converts only the attributes of the object indicated by the detection frame with the largest index value.

7. The information processing method according to any one of claims 1 to 6, wherein when the conversion process is executed, the detection frame of the object whose attribute is not converted in the conversion process is deleted from the captured image.

8. An information processing method according to any one of claims 1 to 7, wherein the setting process sets the detection frame by inputting the acquired captured image into a model that has been trained to set the detection frame for an object that appears in the captured image when the captured image is input.

9. An information processing method according to any one of claims 1 to 8, wherein a plurality of frame images in a time series are acquired as the captured images, the detection frame is set for each object in each frame image in the setting process, and the trajectory of each object is identified based on the change in the position of the detection frame for each object between frame images.

10. The information processing method according to claim 9, further comprising: extracting image features of each object in each frame image from the image area within each detection frame in the plurality of frame images in the time series; calculating the similarity of the image features of each object between different frame images; assigning an identifier indicating the object to each detection frame in each frame image according to the similarity; and identifying the trajectory by tracking the positions of the detection frames assigned the same identifier between frame images.

11. The information processing method according to claim 10, further comprising: estimating a movement position of the detection frame in a next frame image following the plurality of frame images based on the position of the detection frame to which the same identifier is assigned in each of the plurality of frame images in the time series; calculating the distance between the position of the detection frame in the next frame image and the movement position; and assigning the identifier to the detection frame according to the similarity and the distance.

12. An information processing method according to any one of claims 9 to 11, wherein when the trajectory of each object intersects with a virtual line set in the image, it is determined that the object has passed through the virtual line.

13. The information processing method according to claim 12, wherein the total number of objects and / or the number of objects for each attribute that have passed through the virtual line are counted.

14. An analysis device having a control unit, wherein the control unit acquires a captured image, sets a detection frame indicating the coordinate range in which at least a portion of objects appearing in the captured image, including a vehicle, for each object, identifies the attributes of the object indicated by each of the detection frames, and, when the detection frame of the vehicle overlaps with the detection frame of another object, converts the attributes of the other object or vehicle according to the attributes of the vehicle or other object.

15. An imaging system including an imaging device that captures images and an analysis device according to claim 14, wherein the imaging device outputs a plurality of frame images in time series as the captured images to the analysis device, and the analysis device sets the detection frame for each object in each frame image as the setting of the detection frame, identifies the trajectory of each object based on the change in the position of the detection frame for each object between frame images, determines that the object has passed through a virtual line in the image when the trajectory of the object intersects with the virtual line, and counts the total number of objects and / or the number of objects for each attribute that have passed through the virtual line.

16. An analysis system comprising the imaging system of claim 15 and a management device, wherein the analysis device of the imaging system outputs an analysis result including the total number of objects that have passed through the counted virtual line and / or the number for each attribute to the management device, and the management device displays the analysis result and / or an analysis result based on the analysis result on a display device.

Images