Information processing method, analysis device, imaging system, and analysis system

Abstract

To provide an information processing method, etc. for preferably detecting an object.SOLUTION: In an information processing method, a computer executes: processing of acquiring a captured image; processing of setting, for a plurality of objects including a vehicle which is at least some of the objects included in the captured image, detection frames indicating a coordinate range in which the objects exist; processing of identifying the attribute of an object indicated by each of the detection frames; and processing of, when the detection frame of the vehicle overlaps the detection frame of another object, converting the attribute of the other object according to the attribute of the vehicle and vice versa.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method, an analysis device, an imaging system, and an analysis system. [Background technology]

[0002] There are technologies for detecting objects in images. For example, Patent Document 1 discloses a video surveillance system that uses a classifier that has learned the characteristics of people who are carrying belongings or the belongings to determine whether a person in an image captured by a surveillance camera is carrying a specific belonging, determines different attributes of the person depending on the belongings, and outputs the results of statistical processing based on the attribute determination results. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7047944 Summary of the Invention [Problem to be solved by the invention]

[0004] In one aspect, an object is to provide an information processing method and the like that can suitably perform object detection. [Means for solving the problem]

[0005] In one aspect, the information processing method includes a computer executing the following processes: a process of acquiring an image; a process of setting a detection frame indicating the coordinate range in which an object, which is at least a portion of the object shown in the image and includes a vehicle, is located; a process of identifying the attributes of the object indicated by each of the detection frames; and a process of converting the attributes of the vehicle or other object according to the attributes of the other object or vehicle when the detection frame of the vehicle overlaps with the detection frame of another object. [Effects of the Invention]

[0006] In one aspect, object detection can be suitably performed. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of the configuration of an analysis system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of an analysis device. [Figure 3] FIG. 1 is a diagram illustrating an overview of an embodiment. [Figure 4] 10A and 10B are diagrams illustrating an example of attribute conversion processing of a detection frame. [Figure 5] FIG. 10 is a diagram illustrating an object tracking process. [Figure 6] FIG. 10 is a diagram illustrating a process of counting the number of passages. [Figure 7] 10 is a flowchart illustrating an example of a processing procedure executed by the analysis device. DETAILED DESCRIPTION OF THE INVENTION

[0008] (Embodiment) Fig. 1 is a diagram showing an example of the configuration of an analysis system. In this embodiment, an analysis system is described that detects objects (e.g., passersby) from an image captured by a camera (captured image) and tracks the movement of the objects, thereby counting the number of objects (e.g., the number of people) that pass through a specific line or area. The analysis system includes an imaging system 1 and a server 2 (management device). The imaging system 1 and the server 2 are communicatively connected via a network N such as the Internet.

[0009] The imaging system 1 includes an analysis device 10 and a camera 20. The camera 20 is an imaging device that captures images of passersby and the like, and is installed, for example, on a street or inside a building. The analysis device 10 performs object detection and object tracking of objects included in the image captured by the camera 20, and counts the number of objects that pass through a certain line or area based on the results of the detection and tracking. In this embodiment, as an example, the analysis device 10 counts the number of people who pass through a count line (a virtual line) that is virtually set in the area captured by the camera 20 (i.e., the number of people passing through the line).

[0010] Server 2 is a server computer that acquires the analysis results of images obtained by analysis device 10 (in this embodiment, the number of passersby) and manages the analysis results. For example, analysis device 10 counts the number of passersby at predetermined intervals and transmits the analysis results to server 2. At this time, analysis device 10 may send the images captured by camera 20 to server 2, or may discard them without sending them. In this embodiment, analysis device 10 sends only the analysis results to server 2 and discards the captured images. In response to a request from a user, server 2 outputs the analysis results and / or analysis results obtained by further analyzing the analysis results to a display device such as a terminal device (not shown).

[0011] 2 is a block diagram showing an example of the configuration of the analysis device 10. The analysis device 10 includes a control unit 11, a main memory unit 12, a communication unit 13, an input / output unit 14, and an auxiliary memory unit 15. The control unit 11 has one or more processors such as a field programmable gate array (FPGA), a central processing unit (CPU), or a graphics processing unit (GPU), and performs various information processing by reading and executing a program P stored in the auxiliary storage unit 15. The main storage unit 12 is a temporary storage area such as a random access memory (RAM), and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing and transmits and receives information to and from the outside (e.g., the server 2). The input / output unit 14 is an input / output interface that connects the analysis device 10 and the camera 20, acquires captured images from the camera 20, and outputs control signals to the camera 20 for controlling the camera 20. The camera 20 and the analysis device 10 may be connected via a wired or wireless connection.

[0012] When acquiring a captured image from camera 20, analysis device 10 may also acquire time information and / or an identifier of camera 20. This time information and / or identifier may be output from camera 20, or may be identified on the analysis device 10 side. Analysis device 10 may then output the analysis result to server 2 in association with the time information and / or the identifier.

[0013] The auxiliary storage unit 15 is a non-volatile storage area such as a ROM (Read Only Memory), and stores a program P (program product) and other data for the control unit 11 to execute processing. The auxiliary storage unit 15 also stores a detection module 51 and an identification module 52.

[0014] The detection module 51 is a program module used for object detection, and includes one or more machine-learned models (object detection models) that have been trained to detect each object in an input image. More specifically, when an image is input, the detection module 51 identifies a detection frame (bounding box) indicating the coordinate range in which each object in the image may exist, and the object attributes (for example, the class to which the object is predicted to belong) indicated (predicted) by each detection frame, and outputs them in association with each other.

[0015] The output format of the detection frame and attributes is not particularly limited. For example, if the detection frame is rectangular, the position and size of the detection frame may be output as the coordinates of its four corner vertices, or as the coordinates of a vertex and the length and width from that vertex. Furthermore, when identifying the attributes of an object, the detection module 51 may select an attribute that is predicted to be most likely to be the attribute of the object from a predetermined group of attributes and identify it as the attribute of the object.

[0016] The detection module 51 may be, for example, a YOLO (You Only Look Once) program module. The trained model may be a type of CNN (Convolutional Neural Network). The object detection model may be a neural network other than a CNN, or a machine learning model with an architecture other than a neural network. The means for object detection is not limited to machine learning models, and object detection (i.e., setting a detection frame for a captured image and identifying the attributes of an object indicated by the detection frame) may be achieved using, for example, rule-based pattern matching.

[0017] The object detection model may also be composed of multiple trained models. For example, multiple types of object detection models specialized in detecting specific objects may each be configured to set a detection frame and identify attributes, and the results may be combined to provide the object detection result. For example, an object detection model specialized in detecting people as "objects" and an object detection model specialized in detecting automobiles may each be used to set a detection frame and identify attributes, and the combined results may be used as the object detection result.

[0018] The identification module 52 is a program module including one or more trained machine learning models (identification models) used for object tracking. A discrimination model is a model that has been trained to extract image features (image features) when an image is input. The discrimination model of the identification module 52 can be implemented, for example, by DeepSORT. In this embodiment, a machine learning model (identification model) is used to extract image features of a person (object). However, the model may be a model other than DeepSORT, and extraction methods other than machine learning models may also be used. The target from which features are extracted may be an object other than a person, such as a vehicle, or both a person and an object other than a person. A discrimination model may also be composed of multiple trained models. For example, multiple types of discrimination models specialized in extracting image features of specific objects may each extract image features, and the combined results may be used in a downstream process for object tracking. For example, a discrimination model specialized in extracting image features of a person and a discrimination model specialized in extracting image features of a vehicle may each extract image features from the same image, and the combined results may be used in a downstream process for object tracking.

[0019] The analysis device 10 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input and / or a display unit that displays images. The analysis device 10 may also include a reading unit that reads the portable storage medium 1a, and may read and execute the program P from the portable storage medium 1a. In this case, part or all of the detection module 51 and / or the identification module 52 may be stored in the portable storage medium 1a, rather than in the auxiliary storage unit 15.

[0020] 3 is a diagram showing an outline of the embodiment, and the outline of the embodiment will be described with reference to FIG.

[0021] The analysis device 10 sequentially acquires video captured on the street from the camera 20. For example, the analysis device 10 analyzes one minute of video (i.e., a plurality of frame images in a time series for one minute) at predetermined intervals (e.g., every minute), counts the number of people who pass through the virtual line, and transmits (outputs) the number of people and / or analysis results including the number of people to the server 2 as needed. The analysis device 10 may count the number of people who pass through the count line by attribute of the people, or may count the total number of people regardless of attribute. The analysis device 10 may also count both the total number of people who pass through and the number of people who pass by attribute and transmit the counts to the server 2.

[0022] First, analysis device 10 detects objects shown in each frame image of a video captured by camera 20. In this embodiment, analysis device 10 performs object detection using detection module 51. When an image is input, detection module 51 sets a detection frame (so-called bounding box) for each object in the image, and further identifies the attributes of the object indicated (or predicted to be indicated) by the detection frame. Detection module 51 then outputs these in association with each other. In other words, detection module 51 assigns a detection frame to each object in the image, and the detection frame is assigned the attributes of the object indicated by the detection frame. Hereinafter, the "attributes of the object indicated by the detection frame" may also be referred to as the "attributes of the detection frame" or the "attributes assigned to the detection frame."

[0023] The analysis device 10 sequentially inputs frame images to the detection module 51, causing the detection module 51 to set a detection frame for each object in each frame image and identify the attributes of each detection frame. For example, if the detected object is a person, the attribute is identified as "Person," if it is a bicycle, the attribute is identified as "Bicycle," and if it is a car, the attribute is identified as "Car." The attribute "Person" is an example of an attribute indicating a person. Other attributes indicating a person may include more specific attributes such as passerby or store clerk, or attributes that further subdivide age and gender such as male, female, adult, child, and elderly. All of these are examples of "attributes indicating a person." The attributes "Bicycle" and "Car" are also examples of attributes indicating a vehicle. Other attributes indicating a vehicle may include more specific attributes that further subdivide vehicles such as motorcycle, tricycle, light car, and truck. All of these are also examples of "attributes indicating a vehicle."

[0024] Next, the analysis device 10 performs a process to convert the attributes of the detected object. More specifically, the analysis device 10 first determines whether the detection frame of the vehicle and the detection frame of another object overlap. If it determines that they overlap, the analysis device 10 converts the attribute of at least one of the overlapping detection frames to an attribute different from the original attribute. During this attribute conversion, the analysis device 10 converts the attribute of the other detection frame according to (the content of) at least one attribute of each of the overlapping detection frames. In this embodiment, the attribute of the other detection frame is converted according to the attribute of the detection frame of the vehicle.

[0025] Furthermore, when it is determined that three or more detection frames overlap, if at least one of the detection frames satisfies the second condition, the attribute conversion described above may be performed. Furthermore, regarding the second condition, for example, when two vehicle detection frames overlap, the attribute conversion described above may or may not be performed. For example, when a detection frame of a car overlaps with a detection frame of another car, the analysis device 10 may determine that each is a separate car detection frame without performing attribute conversion (and merging or deleting the detection frames, as described below).

[0026] The specific attributes of the vehicle that are converted into the attributes of other objects can be determined as appropriate. For example, these can be determined as appropriate depending on the specifications of the detection module 51 (e.g., the types of objects that can be detected and / or that are the targets of attribute assignment) and how the final object detection results are used.

[0027] In this embodiment, the detection module 51 is assumed to be a model capable of detecting one or more types of vehicles and people as "objects." In this embodiment, an example will be described in which, when a vehicle detection frame and a person detection frame overlap, the attributes of the person detection frame are converted according to the attributes of the vehicle detection frame. However, the objects that the detection module 51 can detect are not limited to this. It is sufficient for the detection module 51 to be able to detect at least some of the objects that appear in the captured image.

[0028] 4A to 4D are diagrams schematically illustrating an example of a detection frame attribute conversion process. In each of the diagrams in Fig. 4A to 4D, a rectangle represents a detection frame, and a word added near the detection frame represents the attribute of the detection frame. As shown in each diagram in Fig. 4, when a vehicle detection frame and a person detection frame overlap, analysis device 10 according to this embodiment converts the attribute representing the person according to the attribute of the vehicle on which the detection frame overlaps.

[0029] In the example of Fig. 4A, when the detection frame of an object whose attribute is "Person" overlaps with the detection frame of an object whose attribute is "Bicycle," the attribute of "Person" is converted to the attribute of "Bicycle" based on the attribute of "Bicycle." In the example of Fig. 4B, when the detection frame of an object whose attribute is "Person" overlaps with the detection frame of an object whose attribute is "Motorcycle," the attribute of the person detection frame is converted to the attribute of "Motorcycle" based on the attribute of "Motorcycle."

[0030] On the other hand, in the example of FIG. 4C, two detection frames of objects with the attribute "person" are superimposed on a detection frame of an object with the attribute "car." In this case, the analysis device 10 may perform attribute conversion determination and processing separately for the detection frames of the two other objects (people) superimposed on the car detection frame, or may process them together. In either case, as in FIGS. 4A and 4B, the attribute "person" is converted to the attribute "car occupant" based on the attribute "car."

[0031] When performing this attribute conversion process, analysis device 10 may determine the converted attributes based on the attributes of the vehicle detection frame (i.e., the vehicle attributes) and the attributes of the object overlapping with the vehicle detection frame (i.e., the attributes of the other object). For example, if detection module 51 is a model capable of detecting one or more types of vehicles and multiple types of objects including people, the attributes of the person detection frame may be converted on the condition that the vehicle detection frame and the person detection frame overlap. In this case, conversion similar to the attribute conversion shown in FIGS. 4A to 4C can be performed.

[0032] Furthermore, when detecting objects other than humans, such as animals such as dogs and cats, and / or structures such as utility poles and signs, the above-described attribute conversion may not be performed if a vehicle detection frame overlaps with a non-human detection frame. For example, if a human detection frame overlaps with a bicycle detection frame, the attribute of the human detection frame may be converted to "Bicycle Rider," whereas if an animal detection frame overlaps with a bicycle detection frame, the attribute of neither detection frame may be converted. Attribute conversion may also be performed when a human detection frame does not overlap with a vehicle detection frame. For example, if a human detection frame does not overlap with a vehicle detection frame, the analysis device 10 may convert the attribute "Person" to the attribute "Walker."

[0033] The analysis device 10 deletes the detection frame of the object whose attribute is not converted (the detection frame of the vehicle in the examples of FIGS. 4A to 4C) at any timing before or after the attribute conversion. As a result, in the examples of FIGS. 4A to 4C, only the person detection frame is extracted and remains, and this remaining person detection frame can be tracked during object tracking, which will be described later. Therefore, when it is desired to count the number of people by object tracking, for example, it is expected that the accuracy of object tracking can be improved and / or the load can be reduced. In this way, according to this embodiment, by leaving only detection frames suitable for subsequent processing, the subsequent processing can be carried out in an appropriate manner.

[0034] Returning to Figure 3, the description continues. Next, the analysis device 10 performs a process of identifying the movement trajectory of each person by tracking the position of the detection frame of the person (object) detected from each frame image between frames. In this embodiment, the analysis device 10 performs object tracking using a Person ReID (Person Re-Identification) technique.

[0035] Fig. 5 is a diagram showing the object tracking process. Fig. 5 illustrates how image features of a person detected from a frame image are extracted, the degree of match (similarity) between the image features of the person and the image features of a person detected in a frame prior to the current frame (for example, the frame immediately prior to the current frame, where object tracking was performed), and the same ID (identifier) ​​is assigned to the same person according to the degree of match.

[0036] For example, analysis device 10 processes a plurality of time-series frame images captured by camera 20 in the order in which they were captured. First, analysis device 10 extracts image features of each person from the image area within the detection frame of each person detected in the first frame image. In this embodiment, analysis device 10 extracts image features of each person by inputting the image of the image area within the detection frame of each person to identification module 52.

[0037] Since the current frame is the first frame image (i.e., the first frame image in the video to be subjected to object detection), analysis device 10 assigns a new ID indicating each person to each detection frame in the current frame. Analysis device 10 stores information such as image feature amounts, coordinates and attributes of each person in a database in association with the ID.

[0038] When processing the next frame, analysis device 10 extracts image features of each person by inputting the image of the image area within the detection frame of each person detected in the current frame to identification module 52. Analysis device 10 then compares the image features of each person in the current frame with the image features of each person detected in the frame before the current frame, and assigns the same ID to the same person.

[0039] Specifically, the analysis device 10 calculates the similarity (e.g., cosine distance) between the image feature of each person in the current frame and the image feature of each person in the frame before the current frame. Note that the "similarity" is not limited to the cosine distance, and for example, cosine similarity may be used. If the calculated similarity is within a threshold, the analysis device 10 determines that the person appearing in the current frame is the same person as the person appearing in the previous frame and assigns the same ID. On the other hand, if the similarity is equal to or greater than the threshold, the analysis device 10 determines that the person appearing in the current frame is a person not appearing in the previous frame and assigns a new ID. The analysis device 10 stores information such as the person's image feature, detection frame coordinates, and attributes in a database in association with the ID.

[0040] Analysis device 10 performs the above process for each frame and assigns an ID identifying each person to the detection frame of each person in each frame. Analysis device 10 then identifies the movement trajectory of the person by tracking the position of detection frames assigned the same ID between frames.

[0041] Analysis device 10 may improve the accuracy of object tracking by considering not only the degree of coincidence (similarity) of image feature amounts within a detection frame, but also the predicted movement position of the detection frame.

[0042] Specifically, analysis device 10 uses a Kalman filter to estimate the movement position of a detection frame in the next frame (current frame) following the previous frames, based on the positions of detection frames assigned the same ID in multiple frames prior to the current frame. Analysis device 10 calculates the distance (e.g., Euclidean distance) between the position of the person detection frame detected in the current frame and the movement position of the detection frame estimated from multiple frames prior to the current frame. Analysis device 10 then assigns an ID indicating a person to each detection frame based on the calculated distance and the similarity of image features.

[0043] Specifically, the analysis device 10 determines whether the calculated distance is within a threshold and whether the similarity is within a threshold. If it is determined that the distance is within the threshold and the similarity is within the threshold, the analysis device 10 determines that the person detected in the current frame is the same person as the person detected in the previous frame, and assigns the same ID to the detection frame. On the other hand, if it is determined that the similarity is within the threshold but the distance is equal to or greater than the threshold, the analysis device 10 determines that the person detected in the current frame is a different person from the person detected in the previous frame, and assigns a new ID to the detection frame. The analysis device 10 identifies the movement trajectory of the person by tracking the positions of detection frames assigned the same ID between frames.

[0044] In this way, analysis device 10 may perform object tracking taking into consideration not only the degree of match of image feature amounts but also the degree of match of the detection frame positions (distance from the predicted movement position).

[0045] Continuing the explanation, returning to Fig. 3, the analysis device 10 then performs processing to count the number of passing people of each attribute and / or the number of passing people (traffic volume) of all people from the movement trajectories of people of each attribute.

[0046] 6 is a diagram showing the process of counting the number of passes, illustrating the movement trajectory of a person and a virtual line (count line) set in the image.

[0047] The virtual line is a line virtually set within the imaging range of the camera 20 and can be arbitrarily set by the user. The virtual line is not limited to a straight line and may be a curved line. The virtual line may not be a two-dimensional line but may be a region with an area (i.e., a virtual region). The analysis device 10 counts the number of passing people for each attribute depending on whether the trajectory of the person for each attribute intersects with the virtual line. Specifically, as shown in FIG. 6, the analysis device 10 identifies the movement trajectory of the person from the position of the detection frame at each time point (frame) and determines for each time point whether the trajectory of the person intersects with the virtual line. For example, the analysis device 10 divides the area in the image into "area 1" and "area 2" using the virtual line and counts the number of people entering "area 2" from "area 1" and the number of people exiting "area 1" from "area 2" (see FIG. 1). The counting may be performed by attribute or the total for all attributes. That is, the analysis device 10 counts the total number of objects (people in this case) that pass through the virtual line and / or the number for each attribute.

[0048] The analysis device 10 transmits the analysis results, including the total count and / or the number for each attribute, to the server 2. At this time, the analysis device 10 may output the analysis results by associating them with time information and / or the identifier of the camera 20. In this embodiment, the analysis device 10 discards the captured images. The server 2 outputs the analysis results in response to a request from a user. For example, the server 2 may display the analysis results on a display device or the like.

[0049] Fig. 7 is a flowchart showing an example of a processing procedure executed by analysis device 10. The processing content executed by analysis device 10 will be described with reference to Fig. 7. For convenience, Fig. 7 shows a series of processing steps from start to finish for a video acquired from camera 20 at one time, but this series of processing steps is repeatedly executed as long as captured images are continuously supplied from camera 20.

[0050] The camera 20 outputs captured images (e.g., video) to the analysis device 10. The camera 20 may send the captured images to the analysis device 10 at any timing of its own device, or may output the captured images at a timing instructed by the control unit 11 of the analysis device 10. The captured images sent from the camera 20 to the analysis device 10 may be accumulated in chronological order in a storage device. The control unit 11 of the analysis device 10 temporarily stores the captured images in an internal or external memory, and the control unit 11 can read and use these in chronological order as appropriate in subsequent processing. Note that if the processing speed of the analysis device 10 is sufficient, the control unit 11 may receive the captured images output from the camera 20 as needed and perform the processing from step S12 onwards, without saving them in a storage device or the like.

[0051] The control unit 11 of the analysis device 10 acquires a captured image (step S11). As mentioned above, "acquiring" here may mean directly acquiring an image from the camera 20, or reading out a captured image that has been stored in a storage device or the like. The control unit 11 acquires one or more frame images.

[0052] The control unit 11 selects one of the multiple frame images in the order in which each frame is captured (step S12). If the control unit 11 itself reads the captured images frame by frame, the read frame image is used as is for subsequent processing, and the processing of step S12 may be omitted. The control unit 11 inputs the selected frame image to the detection module 51, thereby detecting each object in the frame image (step S13). Specifically, the control unit 11 assigns a detection frame (bounding box) to each object in the frame image, which is a detection frame that indicates the range in which the object exists and indicates the attributes of the object.

[0053] When the detection frame of a vehicle overlaps with the detection frame of a person (another object) among the detected objects, the control unit 11 converts the attributes of the person according to the attributes of the vehicle (step S14).The control unit 11 assigns the converted attributes to the person detection frame and deletes the vehicle detection frame from the detection result (step S15).

[0054] The control unit 11 extracts the image area within the detection frame from the frame image and inputs it to the identification module 52 to extract image features of the person (object) that is the subject (step S16). The control unit 11 calculates the similarity between the image features of each person detected in the current frame and the image features of each person detected in the frame before the current frame (step S17). Note that if the current frame is the first frame, the control unit 11 skips the processing of step S17.

[0055] The control unit 11 assigns an ID (identifier) ​​indicating the detected person to each detection frame in the current frame according to the calculated similarity (step S18). Specifically, if the similarity is equal to or less than a threshold, the control unit 11 assigns the same ID as the person detected in the previous frame to the detection frame, and if the similarity is equal to or greater than the threshold, the control unit 11 assigns a new ID to the detection frame. If the current frame is the first frame, the control unit 11 assigns a new ID to the detection frame. The control unit 11 stores information such as image features of the subject person, coordinates and attributes of the detection frame, etc., in association with the ID in the database (step S19). If the control unit 11 has acquired multiple frames (i.e., frames for a predetermined period) in step S11, the control unit 11 repeatedly executes the processes of steps S12 to S19 for the multiple frames. On the other hand, if the control unit 11 has acquired only one frame in step S11, the control unit 11 repeats the processes of steps S12 to S19 for all frames for the predetermined period. The predetermined period mentioned here is set appropriately and is a period equivalent to at least two frames.

[0056] The control unit 11 determines whether the processes of steps S13 to S19 have been performed for all frames for a predetermined period (step S20). If it determines that the processes have not been performed for all frames (S20: NO), the control unit 11 returns the process to step S12. In this case, the control unit 11 selects the next frame image (step S12) and performs the processes of steps S13 to S19.

[0057] If it is determined that processing has been performed on all frames (S20: YES), the control unit 11 identifies the movement trajectory of each person by tracking the position of detection frames assigned the same ID between frame images (step S21). The control unit 11 counts the number of people passing by each attribute by determining whether the trajectory of the person with each attribute intersects with a virtual line in the image (step S22). The control unit 11 outputs (transmits) the analysis result including the counted number of people passing to the server 2 (step S23), and ends the series of processes for the captured image acquired in step S11.

[0058] After the process of S23, or at any timing parallel to steps S11 to S23, the analysis device 10 acquires the next captured image to be processed from the camera 20 (step S11). Then, the process of steps S12 to S23 is continuously executed for this newly acquired captured image. That is, for example, if the camera 20 is capturing video in real time and outputting it to the analysis device 10, the process shown in Fig. 7 loops to step S11 after the process of step S23 (or step S12 if a captured image has been acquired earlier), and is repeated for the new captured image.

[0059] On the other hand, if the camera 20 intermittently captures video or still images and sends them to the analysis device 10, the analysis device 10 may perform processing from step S11 onwards when it acquires an image from the camera 20 (or stores the image in a memory device), and after the processing is completed, it may wait until the next image is input from the camera 20 (or stored in a memory device).

[0060] Analysis device 10 may perform the processes of steps S13 to S19 for all frame images of the acquired moving image, or may perform the processes of steps S13 to S19 for frames extracted at any interval from all frames, thereby determining that all frames have been processed. For example, if a 10-second moving image with a frame rate of 5 fps is acquired in step S11, analysis device 10 may perform the processes of steps S13 to S19 a total of 10 times, every 5 frames, to determine that all frames of this 10-second moving image have been processed in step S20.

[0061] 7, the processing from step S16 onwards does not necessarily require the direct use of the frame image selected in step S12. Therefore, control unit 11 may discard the frame image that has undergone the processing of steps S13 to S16 as appropriate. Furthermore, the processing of steps S16 to S19 for a certain frame image (referred to as the first frame image) and the processing of steps S13 to S15 for the frame image selected as the next processing target (referred to as the second frame image) may be executed in parallel. In other words, while processing related to object tracking is being executed for the first frame image, processing related to object detection may be executed for the second frame image.

[0062] Furthermore, if the captured images output at one time are multiple frame images, control unit 11 may divide the multiple frame images into predetermined periods and perform the processes of steps S12 to S19 for the video for that predetermined period. Then, once these processes are completed, control unit 11 may determine in S20 that the processes have been performed for all frames for that predetermined period. For example, if the video from camera 20 is five minutes long, control unit 11 may determine YES in step S20 after completing the processes of steps S12 to S19 for one minute of the video, and then perform the processes of steps S21 to S23. This process may be repeated five times.

[0063] As described above, according to this embodiment, the detection frame of a vehicle and the detection frame of a person (other object) riding in the vehicle are integrated, and an attribute according to the vehicle is assigned to the person. This makes it possible to appropriately determine the attribute of the object indicated by the detection frame during object detection. Furthermore, by integrating or deleting one or more of the superimposed detection frames, it is possible to extract a detection frame suitable for subsequent processing (for example, object tracking and headcount, measurement of the number of people and / or vehicles passing by, etc.). In other words, it can be said that object detection can be performed suitably. Furthermore, after object detection, object tracking, headcount, measurement of the number of people passing by, etc. can be performed suitably.

[0064] (Variation) In the above-described embodiment, an example was described in which, when a vehicle detection frame and a person detection frame overlap, the attributes of the person are converted. However, when a vehicle detection frame and a person detection frame overlap, the analysis device 10 according to this embodiment may convert the attributes of the object indicated by the vehicle detection frame, leave the vehicle detection frame, and delete the person detection frame. Referring to the example of FIG. 4A , for example, the analysis device 10 may convert the attribute of the “bicycle” detection frame to “bicycle with person on board” and delete the “person” detection frame. In this way, the analysis device 10 only needs to be able to convert the attributes of the person or vehicle according to the attributes of the vehicle or person. In this case, as in the above-described example, a detection frame suitable for object detection can be extracted, thereby enabling suitable object detection. Furthermore, after object detection, object tracking, counting people, measuring traffic volume, and the like can be performed suitably.

[0065] Furthermore, after attribute conversion, analysis device 10 may integrate multiple overlapping detection frames and assign the converted attributes to the integrated detection frames, rather than deleting detection frames with attributes that were not converted. For example, analysis device 10 may set only the overlapping portion of the vehicle detection frame and the person detection frame as a new detection frame, or may set a new detection frame that includes the vehicle detection frame and the person detection frame. When detection frames are integrated in this way, the integrated detection frame is no longer a frame that indicates the presence area of ​​a single object, and therefore the converted attributes may be assigned to the integrated detection frame. As such, various methods are possible for integrating detection frames.

[0066] Furthermore, when deleting (or merging) detection frames and / or converting attributes, the analysis device 10 may perform the merging (or deletion) of detection frames and / or attribute conversion by referring to the object detection results in previous (preferably, the most recent one or more) frame images. For example, if attribute conversion is performed by referring only to the determination results of a single frame, detection frames may be merged even when a pedestrian happens to pass in front of a bicycle, and the pedestrian's attribute may be converted to "bicycle rider." Therefore, the analysis device 10 may prevent such unintended attribute conversion and deletion (or merging) of detection frames by referring to whether a vehicle detection frame and a person detection frame overlap in other frames (preferably, frames processed immediately before and / or after in steps S13 to S19). For example, if the detection frame of an object assigned the attribute "bicycle" moves between frames, the analysis device 10 converts the attribute of the object to "motorcycle rider."

[0067] Specifically, when the analysis device 10 determines whether a vehicle detection frame and a person detection frame overlap in each frame, it stores the position or identifier of the person detection frame and the determination result. When the analysis device 10 determines that a vehicle detection frame and a person detection frame overlap in one frame, it reads the determination result for the other frames. The analysis device 10 then determines whether a detection frame having the same position or identifier as the person detection frame overlaps with the vehicle detection frame in a predetermined number or more (e.g., more than half) of frames. When it determines that there is overlap in a predetermined number or more frames, the analysis device 10 converts the person attributes according to the vehicle attributes and deletes the vehicle detection frame. On the other hand, when it determines that there is no overlap in a predetermined number or more frames, the analysis device 10 does not convert the attributes and does not delete the vehicle detection frame. This makes it possible to prevent unintended deletion or merging of detection frames.

[0068] Furthermore, when determining whether or not detection frames overlap, the analysis device 10 may calculate an index value of the degree of overlap between the detection frame of the vehicle and the detection frame of another object, and refer to this to determine whether or not to perform attribute conversion. For example, the analysis device 10 may convert the attribute when the aforementioned index value is a percentage equal to or greater than a threshold, and may not convert the attribute when the index value is less than the threshold.

[0069] An example of an index value for the degree of overlap is Intersection over Union (IoU). In this case, the analysis device 10 may convert the attributes of detection frames whose IoU with the vehicle detection frame is equal to or greater than a predetermined threshold, and may not convert the attributes of detection frames whose IoU is less than the threshold even if they overlap.

[0070] Another example of an index value for the degree of overlap may be the "proportion of the area where the detection frame of the vehicle overlaps with the detection frame of another object to the total area of ​​the detection frame of the vehicle." Here, the index value for this proportion is referred to as "IoV (Intersection over Vehicle)." When IoV is used as the index value, it is also possible to determine whether or not to perform attribute conversion using a processing procedure similar to that for IoU described above.

[0071] Another example of an index value for the degree of overlap is the "proportion of the area overlapping with the vehicle detection frame to the entire area of ​​the person detection frame." Here, the index value for this proportion is referred to as "IoP (Intersection over Person)." When IoP is employed, the analysis device 10 identifies whether the detection frames of other objects overlapping with the vehicle detection frame are human detection frames during attribute conversion. That is, the analysis device 10 refers to the attributes of the other objects overlapping with the vehicle detection frame. The subsequent processing is the same as the IoU and IoV described above, even when IoP is employed as the index value.

[0072] When using such an index value of the degree of overlap for judgment, only the attribute of the object indicated by the detection frame with the largest index value may be converted. For example, suppose three person detection frames overlap with a vehicle detection frame. In this case, the analysis device 10 converts the attribute of only one of the three person detection frames with the largest index value of the degree of overlap (e.g., IoU, IoV, or IoP), and does not need to convert the attributes of the remaining two person detection frames. Note that if the index values ​​for two or more detection frames are the same, the attributes of all detection frames with the same value may be converted. Specifically, in the example of FIG. 4C, the index values ​​of the two "Person" detection frames are the same, so the attribute of both detection frames may be converted to "Occupant."

[0073] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0074] The features described in each embodiment can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations regardless of the form of reference. [Explanation of symbols]

[0075] 1. Imaging system 10 Analyzer 11 Control section 12 Main memory 13 Communications Department 14 Input / output section 15 Auxiliary storage P Program 51 Detection Module 52 Identification Module 20 Camera 2 Server (management device)

Claims

1. A process of acquiring a captured image; a process of setting a detection frame indicating a coordinate range in which at least some of the objects appearing in the captured image, the detection frame indicating a coordinate range in which the objects appear ... including a vehicle, are present; A process of identifying attributes of objects indicated by each of the detection frames; When the detection frame of the vehicle and the detection frame of another object overlap, a process of converting the attributes of the other object or vehicle according to the attributes of the vehicle or the other object. An information processing method performed by a computer.

2. In the conversion process, the attributes of the other object are converted according to the attributes of the vehicle. The information processing method according to claim 1 .

3. In the conversion process, the attributes of the other object are converted according to the attributes of the vehicle and the attributes of the other object. The information processing method according to claim 2 .

4. In the conversion process, the attributes of the vehicle are converted according to the attributes of the other object. The information processing method according to claim 1 .

5. In the conversion process, the attribute is converted when an index value of the degree of overlap between the detection frame of the vehicle and the detection frame of the other object is equal to or greater than a threshold, and the attribute is not converted when the index value is less than the threshold. The information processing method according to any one of claims 2 to 4.

6. In the conversion process, only the attribute of the object indicated by the detection frame with the largest index value is converted. The information processing method according to claim 5 .

7. When the conversion process is executed, the detection frame of the object whose attribute is not converted in the conversion process is deleted from the captured image. The information processing method according to any one of claims 1 to 6.

8. In the setting process, the captured image is input to a model that has been trained to set the detection frame for an object appearing in the captured image when the captured image is input, thereby setting the detection frame. The information processing method according to any one of claims 1 to 7.

9. acquiring a plurality of frame images in time series as the captured images; In the setting process, the detection frame is set for each object in each frame image; Identifying the trajectory of each object based on the transition of the position of the detection frame of each object between frame images. The information processing method according to any one of claims 1 to 8.

10. extracting image features of each object in each frame image from an image region within each of the detection frames in the plurality of frame images in the time series; Calculate the similarity of the image features of each object between different frame images, assigning an identifier indicating the object to each of the detection frames in each frame image according to the degree of similarity; The trajectory is identified by tracking the positions of the detection frames to which the same identifier is assigned between frame images. The information processing method according to claim 9.

11. estimating a movement position of the detection frame in a next frame image following the plurality of frame images based on positions of the detection frame to which the same identifier is assigned in each of the plurality of frame images in the time series; calculating a distance between the position of the detection frame in the next frame image and the movement position; The identifier is assigned to the detection frame according to the similarity and the distance. The information processing method according to claim 10.

12. When the trajectory of each object intersects with a virtual line set in the image, it is determined that the object has passed through the virtual line. The information processing method according to any one of claims 9 to 11.

13. Counting the total number of objects passing through the virtual line and / or the number of objects for each attribute. The information processing method according to claim 12.

14. An analyzer including a control unit, The control unit Acquire a captured image, a detection frame indicating a coordinate range in which each object, including a vehicle, exists, is set for at least a portion of objects shown in the captured image; Identifying attributes of objects indicated by each of the detection frames; When the detection frame of the vehicle and the detection frame of another object overlap, the attribute of the other object or the vehicle is converted according to the attribute of the vehicle or the other object. Analyzer.

15. An imaging system including an imaging device that captures an image and the analysis device according to claim 14, The imaging device is outputting a plurality of time-series frame images as the captured images to the analysis device; The analysis device As the setting of the detection frame, the detection frame is set for each object in each frame image; Identifying a trajectory of each object based on a transition of the position of the detection frame of each object between frame images; When the trajectory of each object intersects with a virtual line in the image, it is determined that the object has passed through the virtual line; Counting the total number of objects passing through the virtual line and / or the number of objects for each attribute. Imaging system.

16. The imaging system according to claim 15; An analysis system including: The analysis device of the imaging system outputting an analysis result including the total number of objects that have passed through the counted virtual line and / or the number for each attribute to the management device; The management device The analysis results and / or analysis results based on the analysis results are displayed on a display device. Analysis system.

Images