Information processing device, identity determination method, and storage medium

Abstract

Technical problem: Improve the accuracy of determining the identity of detected objects. Solution: The information processing device (1) includes: a replacement unit (103) that replaces the values ​​of the vertical width and horizontal width of the object detected by the object detection unit (101); and an identity determination unit (104) that determines whether the objects detected by the object detection unit (101) from multiple images are the same object based on the replaced vertical width and horizontal width values.

Description

Technical Field

[0001] This invention relates to an information processing apparatus for detecting objects from images, etc. Background Technology

[0002] In recent years, due to the development of machine learning such as deep learning, the accuracy of object recognition / detection in images has been improving. Consequently, the applications of image recognition technology are expanding. However, since the current detection accuracy is not 100%, further improvements are needed to expand its applications. When using deep learning (or other neural network system detection techniques, etc.) to detect objects in images, the first step is to prepare teaching data. Teaching data is created by associating the image with detailed information about the objects presented in the image (e.g., object identifiers, object position, size, shape, angle, etc.). Then, machine learning is performed using the teaching data, outputting one or more weights (learned model) as the learning result. Finally, object detection is performed using this learned model.

[0003] Furthermore, when performing object detection based on dynamic images, it is determined whether an object detected in a certain frame of the dynamic image is the same object detected in frames before and after that frame. For example, in Patent Document 1 below, the detection (tracking) of moving objects and people is based on the differences in the position and direction of movement of objects and people between current and past frames.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Publication No. 2010-257441 (published on November 11, 2010) Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] In object detection using machine learning algorithms, it is possible to detect objects and also to detect the size and rotation angle of the detected objects. Moreover, size and rotation angle can be used to determine the identity of objects detected in different frames.

[0009] However, even if the object being detected is the same, when using different learned models with different learning settings (teaching data, parameter settings, etc.), the orientation of the axis used as the reference for calculating the rotation angle may change. Furthermore, this can lead to differences in the detected rotation angle and the object's longitudinal / lateral width.

[0010] In this regard, based on Figure 16 Please provide an explanation. Figure 16 This diagram illustrates examples where the same object is detected in different ways due to different learned models used for object detection, resulting in different rotation angles. Figure 16 In the example, object detection is performed from each of images 1000 and 1100, which are extracted from a moving image of object 1010. The object 1010 presented in image 1000 and the object 1010 presented in image 1100 are the same object, but rotated over time (image 1000 was taken earlier).

[0011] Object detection is performed using learned models a and b. Both models a and b are designed for detecting objects 1010, but they were built using machine learning with different learning settings (teaching data, parameter settings, etc.). Furthermore, in Figure 16 In the example, the object detection results of the learned model a for images 1000 and 1100 are shown as bounding boxes 1020a and 1120a. Such bounding boxes can be drawn using information output by the learned model a that indicates the location of the detected object (e.g., the coordinates of the center position).

[0012] Box 1020a has a vertical width of 32, a horizontal width of 10, and an angle of -9°. Box 1120a has the same vertical and horizontal widths as box 1020a, and an angle of 7°. Furthermore, the units for the width and height are arbitrary. Additionally, for object classification, since the learned model is also used, classification results are output.

[0013] When determining whether objects detected from images 1000 and 1100 are the same object based on the object detection results using these learned models a, they are accurately identified as the same object. This is because the width and height of boxes 1020a and 1120a are equal, and the absolute value of the angle difference is a sufficiently small value of 16° (7° + 9°). Furthermore, a threshold set based on the frame rate (e.g., 10 fps: Frames Per Second) of the extraction source of images 1000 and 1100, i.e., the dynamic image, can be used as a benchmark to determine whether the angle difference is small.

[0014] Additionally, the object detection results of the learned model b for images 1000 and 1100 are shown as bounding boxes 1020b and 1120b. Bounding box 1020b is the same as bounding box 1020a, with a vertical width of 32, a horizontal width of 10, and an angle of -9°. On the other hand, bounding box 1120b is different from bounding box 1120a, with a vertical width of 10, a horizontal width of 32, and an angle of 97°.

[0015] Boxes 1120a and 1120b have the same appearance, but their vertical and horizontal dimensions are opposite, and their angles are also offset by 90°. Therefore, if the object detection results based on the learned model b are used to determine whether the objects detected in images 1000 and 1100 are the same object, they will be mistakenly identified as not being the same object. This is because the widths of boxes 1020b and 1120b are different, and the absolute value of the angle difference is as large as 106° (97° + 9°), exceeding the threshold. Furthermore, this problem is not limited to deep learning; it can also occur in object detection using other learned models.

[0016] One objective of the present invention is to provide an information processing device or the like that can improve the accuracy of determining the identity of objects detected in multiple images over a time sequence.

[0017] (II) Technical Solution

[0018] To address the aforementioned technical problems, an information processing apparatus according to one aspect of the present invention includes: an object detection unit that detects an object from each of a plurality of images captured in a time sequence, and detects the longitudinal width and lateral width of the detected object; an identity determination unit that determines whether a first object detected from a first image of the plurality of images and a second object detected from a second image of the plurality of images are the same object; and a replacement unit that replaces the values ​​of the longitudinal width and lateral width of the first object or the second object, wherein when the replacement unit performs the replacement, the identity determination unit performs the determination based on the replaced values ​​of the longitudinal width and lateral width.

[0019] To address the aforementioned technical problems, one aspect of the present invention provides an identity determination method executed by one or more information processing devices, comprising: an object detection step, detecting an object from each of a plurality of images captured in a time sequence, and detecting the longitudinal width and lateral width of the detected object; a replacement step, replacing the values ​​of the longitudinal width and lateral width of a first object detected from a first image of the plurality of images or a second object detected from a second image of the plurality of images; and an identity determination step, determining whether the first object and the second object are the same object based on the replaced values ​​of the longitudinal width and lateral width.

[0020] (3) Beneficial effects

[0021] According to one aspect of the present invention, the accuracy of determining the identity of objects detected in multiple temporal images can be improved. Attached Figure Description

[0022] Figure 1This is an example of a functional block diagram of a control unit of an information processing apparatus according to an embodiment of the present invention.

[0023] Figure 2 This is a block diagram showing an example of the structure of a foreign matter detection system including the above information processing apparatus.

[0024] Figure 3 This is a diagram showing a situation where a garbage collection vehicle drops garbage into a garbage pit in a garbage incineration facility.

[0025] Figure 4 This is a diagram showing the inside of the garbage pit.

[0026] Figure 5 This is a diagram for explaining an image captured by a garbage imaging device.

[0027] Figure 6 This is a flowchart showing an example of a process for detecting foreign matter from a garbage image.

[0028] Figure 7 This is a flowchart showing an example of an object detection process for a still image.

[0029] Figure 8 This is a diagram showing an example of a data structure when saving object detection results for a still image.

[0030] Figure 9 This is a diagram showing an example of a rooted graph (in Japanese: 根付きグラフ).

[0031] Figure 10 This is a diagram showing an example of a data structure when saving information related to a rooted graph.

[0032] Figure 11 This is a flowchart showing an example of a node creation process.

[0033] Figure 12 This is a flowchart showing an example of a node connection process.

[0034] Figure 13 This is a flowchart showing an example of an object detection process for a moving image.

[0035] Figure 14 This is a diagram for explaining a process for visual confirmation.

[0036] Figure 15 This is a diagram for explaining a process for displaying an image of a foreign matter.

[0037] Figure 16 This is a diagram showing an example where the same object is detected at different rotation angles due to different learned models for object detection.

[0038] Explanation of reference numerals

[0039] 1-Information processing device; 101-Object detection unit; 103-Replacement unit; 104-Identity determination unit; 105-Motion image object detection unit; 106-Selection display control unit (display control unit); 4-Selection display device (display device). Detailed Implementation

[0040] In recent years, there has been a problem of improperly added materials (hereinafter referred to as improper materials) being introduced into waste incineration facilities. Due to the introduction of improper materials, problems such as poor combustion in the incinerator and blockages at the ash removal equipment have occurred, sometimes even requiring an emergency shutdown of the incinerator. Currently, staff at waste incineration facilities randomly select collected waste and manually check whether it contains improper materials, placing a heavy burden on the operators.

[0041] Furthermore, to reduce the amount of inappropriate waste transported to waste incineration facilities, a system is needed that can detect inappropriate waste from the transported waste and alert the collection manager when necessary. In this case, it is undesirable to label waste that is not actually inappropriate as such. Additionally, directly showing the manager the captured images is not preferable because it is difficult to determine the exact time and location at which the inappropriate waste was photographed.

[0042] An information processing apparatus 1 according to one embodiment of the present invention can solve the above-mentioned problems. The information processing apparatus 1 has the function of detecting inappropriate substances from waste transported into a waste incineration facility. Specifically, the information processing apparatus 1 uses images of waste being disposed of in a landfill to detect inappropriate substances. Furthermore, regarding the landfill, it will be based on... Figure 4 This will be explained below. Additionally, inappropriate items can be detected after the garbage has been disposed of. Furthermore, inappropriate items are those that should not be burned in the incinerator of a waste incineration facility. Specific examples of inappropriate items will be explained below.

[0043] [System Structure]

[0044] based on Figure 2 The structure of the improper object detection system of this embodiment will be described. Figure 2 This is a block diagram illustrating a structural example of an improper object detection system 100. The improper object detection system 100 includes an information processing device 1, a garbage collection device 2, a vehicle information collection device 3, a selection display device 4, and an improper object display device 5.

[0045] In addition, Figure 2The figure also shows an example of the hardware structure of the information processing device 1. As shown, the information processing device 1 includes: a control unit 10, a high-speed storage unit 11, a large-capacity storage unit 12, an image IF (interface) unit 13, a vehicle information IF unit 14, a selection display IF unit 15, and an object display IF unit 16. The information processing device 1 can be, for example, a personal computer, a server, or a workstation.

[0046] Each part of the overall control information processing device 10 in the control unit 10. Based on Figure 1 The functions of each part of the control unit 10 described below can be implemented either by logic circuits (hardware) formed in integrated circuits (IC chips) or by software. This software may also include an identity determination program that enables the computer to function as the object detection unit, replacement unit, and identity determination unit described below. In the case of software implementation, the control unit 10 may be configured as, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a combination thereof. Furthermore, in this case, the software is pre-stored in the large-capacity storage unit 12. The control unit 10 then loads the software into the high-speed storage unit 11 and executes it.

[0047] Both the high-speed storage unit 11 and the large-capacity storage unit 12 are storage devices for various types of data used by the information processing device 1. Compared to the large-capacity storage unit 12, the high-speed storage unit 11 is a storage device capable of writing and reading data at high speed. Compared to the high-speed storage unit 11, the large-capacity storage unit 12 has a larger data storage capacity. As the high-speed storage unit 11, high-speed access memory such as SDRAM (Synchronous Dynamic Random-Access Memory) can also be used. As the large-capacity storage unit 12, it can also be used with HDD (Hard Disk Drive), SSD (Solid-State Drive), SD (Secure Digital) card, or eMMC (embedded Multi-Media Controller).

[0048] Image IF unit 13 is an interface for communicating between the garbage collection device 2 and the information processing device 1. Additionally, vehicle information IF unit 14 is an interface for communicating between the vehicle information collection device 3 and the information processing device 1. These IF units can be either wired or wireless communication IF units. For example, USB (Universal Serial Bus), LAN (Local-Area Network), or wireless LAN can also be used as these IF units.

[0049] The selection display IF section 15 is an interface for enabling communication between the selection display device 4 and the information processing device 1. Additionally, the non-functional display IF section 16 is an interface for enabling communication between the non-functional display device 5 and the information processing device 1. These IF sections can be either wired or wireless communication IF sections. For example, HDMI (High-Definition Multimedia Interface, registered trademark), DisplayPort, DVI (Digital Visual Interface), VGA (Video Graphics Array), S-Video, or RCA terminals can be used as these IF sections.

[0050] The garbage-capturing device 2 captures images of garbage being dumped into the garbage pit and sends the captured images to the information processing device 1. Hereinafter, these captured images will be referred to as garbage images. As an example, the garbage-capturing device 2 may also be a high-speed shutter camera that captures moving images. Furthermore, the garbage images can be either moving images or sequential still images captured in succession. The garbage images are input to the information processing device 1 via the image IF unit 13. Moreover, the input garbage images can be processed directly by the control unit 10, or they can be stored in the high-speed storage unit 11 or the large-capacity storage unit 12 and then processed by the control unit 10.

[0051] Vehicle information collection device 3 delivers garbage and collects identification information of the vehicles (so-called garbage collection trucks) that dump the garbage into the garbage pit, sending it to information processing device 1. Furthermore, the following will be based on... Figure 4 This section explains how garbage is disposed of in a garbage pit using a garbage collection truck. This identification information is used by the vehicle identification unit 107 to determine the subject of the garbage delivery. This identification information can be, for example, information indicating the license plate number. In this case, the vehicle information collection device 3 can photograph the license plate and send the photographed image as identification information to the information processing device 1. Alternatively, the vehicle information collection device 3 can accept the input of identification information from the garbage collection truck 200 and send it to the information processing device 1.

[0052] The selection display device 4 displays an image of the inappropriate object detected by the information processing device 1. In the inappropriate object detection system 100, considering that the information processing device 1 might mistakenly identify non-inappropriate waste as inappropriate, the selection display device 4 displays an image of the inappropriate object detected by the information processing device 1, and the user visually confirms whether the object presented in the image is inappropriate. Then, the person responsible for visual confirmation selects the image containing the inappropriate object from the images displayed on the selection display device 4.

[0053] The improper item display device 5 displays the image of the improper item detected by the information processing device 1, which is selected by the selection display device 4, thus visually confirming the presence of the improper item. The improper item display device 5 displays the image to draw the attention of the person in charge of transporting the improper item, the operator, etc.

[0054] [Photography of garbage]

[0055] Figure 3 This diagram shows a garbage collection truck 200 dumping garbage into a garbage pit at a waste incineration facility. Figure 4 This is a diagram showing the interior of a landfill pit. A landfill pit is a temporary storage area for waste collected at a waste incineration facility; the waste in the pit is then sequentially fed into the incinerator for combustion. For example... Figure 3 As shown, the waste incineration facility is equipped with multiple doors, including doors 300A and 300B (hereinafter, unless there is a need to distinguish between the various doors, they will be collectively referred to as door 300). Additionally, as... Figure 4 As shown, a garbage pit is provided in front of the door 300. That is, by opening the door 300, a disposal opening for throwing garbage into the garbage pit is created. Figure 3 As shown, the garbage collection truck 200 dumps garbage into the garbage pit from the dumping port.

[0056] Garbage photography device 2 is installed on the side capable of photographing garbage along the edge. Figure 4 The location of the flowing waste on the 600-degree inclined plane. For example, it can be... Figure 3 and Figure 4 The garbage-detecting device 2 is installed at the installation position 400 shown. Since the installation position 400 is located on the surface of each door 300, when the garbage-detecting device 2 is installed at the installation position 400, it is positioned above the inclined plane 600 when the door 300 is open, a position suitable for detecting garbage. Of course, the installation position of the garbage-detecting device 2 can be set to any position where it can detect garbage flowing along the inclined plane 600.

[0057] Alternatively, if the vehicle information collection device 3 is a camera, it can also be installed at mounting position 400. When the garbage truck 200 approaches the door 300, since the door 300 is closed, the vehicle information collection device 3 installed at mounting position 400 can capture images of the garbage truck 200's license plate, etc. Of course, the installation position of the vehicle information collection device 3 can be set to any location where it can capture images of the garbage truck 200, or it can be installed at a different location than the garbage camera 2. Furthermore, the vehicle information collection device 3 can be, for example, an information input device. In this case, the structure could be such that the vehicle information collection device 3 is installed in the operator's cab and accepts input of identification information of the garbage truck 200 from the operator.

[0058] Figure 5 This diagram illustrates images captured using the garbage collection device 2. Image 610 in this diagram is a composite image obtained by extracting frame images 610a-610h from a dynamic image capturing the slide of a board 620 (as an object) sliding down an inclined plane 600. Figure 5 The plates 620 that are presented in each frame of the image 610a to 610h are respectively set to 620a to 620h. In addition, the up and down direction in the image is set as the Y-axis direction, the upward direction is set as the positive direction of the Y-axis, the left and right direction is set as the X-axis direction, and the rightward direction is set as the positive direction of the X-axis.

[0059] In addition, Figure 5 The lower left corner represents frame image 610a, and the lower right corner represents frame image 610f. Both the plate 620a presented in frame image 610a and the plate 620f presented in frame image 610f are images of the same plate 620, but their positions, rotation angles, and sizes differ. The reason for the size difference is that the inclined plane 600 is tilted, and the further downstream the inclined plane 600 is, the further away it is from the garbage collection device 2. As will be explained in detail below, the information processing device 1 can take into account these changes in position, rotation angle, and size to determine the identity of the plates 620a to 620h detected from frame images 610a to 610h.

[0060] Furthermore, the plate 620d in image 610 appears washed out. When frame image 610d is captured, if strong light shines on the plate 620 and its surroundings, this area becomes washed out, making it impossible to detect the plate 620 from such an image. The same applies when the plate 620 is temporarily hidden by other debris, etc., making it impossible to detect the plate 620 from a portion of the frame image. As will be explained in detail below, according to the information processing device 1, even if an object cannot be detected from a portion of the frame image, it can still be accurately detected as long as it can be detected from the preceding and following frame images.

[0061] [Device Structure]

[0062] based on Figure 1 The structure of information processing device 1 will be described. Figure 1 This is an example of a functional block diagram of the control unit 10 of the information processing device 1. Figure 1 The control unit 10 shown includes an object detection unit 101, a curve generation unit 102, a replacement unit 103, an identity determination unit 104, a moving image object detection unit 105, a selection display control unit 106, an incoming vehicle determination unit 107, and an inappropriate object display control unit 108.

[0063] The object detection unit 101 detects objects from each of a plurality of images captured in a time sequence. Furthermore, the object detection unit 101 also detects object information of the detected objects. This object information includes information indicating the vertical and horizontal widths of the detected object. In addition, the object information may also include information indicating the rotation angle of the detected object, the position information of the detected object, and classification information indicating the category of the detected object.

[0064] The object detection unit 101 can detect objects using a machine learning model, such as a neural network. In this case, the structure simply needs to pre-store weight values ​​calculated using machine learning in a large-capacity storage unit 12, and then read these weight values ​​when the object detection unit 101 detects an object. This machine learning uses images of various inappropriate objects as training data. Furthermore, in addition to inappropriate objects, similar objects with similar appearances can also be learned as detection targets. This improves the accuracy of inappropriate object detection. Additionally, during machine learning, it can also learn to classify detected objects.

[0065] When using the learned model as described above, the object detection unit 101 generates object information of the objects detected in the image based on the output value obtained by inputting the garbage image into the learned model, and outputs it. Additionally, the object detection unit 101 outputs a probability value representing the accuracy of the detection result.

[0066] The curve generation unit 102 generates a root image based on the detection results of the object detection unit 101. The root image will be explained in detail below. It is generated based on the object detection results of the object detection unit 101 on each frame of the image extracted from the dynamic image.

[0067] The replacement unit 103 replaces the values ​​of the longitudinal width and transverse width of one of the two objects that are the objects of the identity determination unit 104. When the replacement unit 103 performs this replacement, the identity determination unit 104 determines the identity based on the replaced longitudinal width and transverse width values.

[0068] The identity determination unit 104 determines whether the object detected by the object detection unit 101 from one of the multiple images in the time sequence and the object detected from another of the multiple images in the time sequence are the same object.

[0069] The moving image object detection unit 105 detects objects from a moving image using the root map generated by the curve generation unit 102. Unlike the object detection unit 101, which detects objects from a single image, the moving image object detection unit 105 detects objects presented in a moving image based on object detection results from multiple images; these detection units differ in this respect. Details regarding object detection in moving images will be explained below.

[0070] The selection display control unit 106 causes the selection display device 4 to display an image of an object detected by the moving image object detection unit 105. The person in charge of visual verification checks whether an inappropriate object is present in the displayed image and selects an image containing the inappropriate object. Furthermore, the selection display control unit 106 accepts the image selection made by the person in charge of visual verification.

[0071] The vehicle identification unit 107 uses the identification information received from the vehicle information collection device 3 to identify the vehicle (e.g., the vehicle that received the waste) that received the waste. Figure 3 (Garbage collection vehicle 200). Furthermore, when the information processing device 1 detects an inappropriate object in the garbage previously transported by the transport vehicle as determined by the transport vehicle determination unit 107, the inappropriate object display device 5 displays an image of the inappropriate object. This allows the person in charge of transporting garbage via the transport vehicle to be alerted to the image of the inappropriate object.

[0072] As described above, the information processing apparatus 1 includes: an object detection unit 101 that detects an object from each of a plurality of images captured in a time sequence, and detects the vertical width and horizontal width of the detected object; an identity determination unit 104 that determines whether an object detected from one of the plurality of images is the same object as an object detected from another of the plurality of images; and a replacement unit 103 that replaces the values ​​of the vertical width and horizontal width of the two objects. Furthermore, when the replacement unit 103 performs the aforementioned replacement, the identity determination unit 104 determines the identity based on the replaced values ​​of the vertical width and horizontal width. Therefore, even when the axes serving as the references for determining the vertical and horizontal widths of an object are offset by 90°, the identity of the object can be accurately determined, thus improving the accuracy of determining the identity of objects detected in multiple images in a time sequence.

[0073] [Overall processing flow]

[0074] Figure 6This is a flowchart illustrating an example of a process for detecting inappropriate items from garbage images. As a prerequisite for this process, it is assumed that the garbage images captured by the garbage imaging device 2 are stored in the large-capacity storage unit 12. Furthermore, it is assumed that the aforementioned files are moving image files. Alternatively, multiple sequential still image files can be used instead of moving image files. In this case, the still image files used can be managed, for example, through a file list stored in the large-capacity storage unit 12. Additionally, the still image files used can be compiled and stored in a designated directory within the large-capacity storage unit 12. In the following description, moving image files and still image files may be simply labeled as dynamic images and still images.

[0075] In S200, information processing device 1 performs object detection processing on the still image. The following will be based on... Figure 7 In detail, in the object detection process, the object is detected based on all still images extracted from the above dynamic images, that is, the full-frame images.

[0076] Information processing device 1 performs node creation processing in S300 and node connection processing in S600. Based on these processes, nodes are created... Figure 9 The root graph shown below. The following will be based on... Figure 11 This section details the node creation and processing process, and is based on... Figure 12 This section details the node connection process.

[0077] In S800, information processing device 1 performs object detection processing on dynamic images. The following will be based on... Figure 13 The details are explained below, and the root diagram mentioned above is used in the S800 decision. The S800 processing ends, thus... Figure 6 The entire process is now complete.

[0078] In addition, Figure 6 In the processing, steps S200, S300, S600, and S800 are processed sequentially (serially). That is, the entire moving image file is verified in S200, then a root map is created in S300 and S600, and finally, the root map is used to detect objects in S800. When object detection is performed from the image being captured, rather than from a completed moving image, the internal processing of steps S200, S300, S600, and S800 can be performed in parallel. Detailed explanations of the parallel processing are omitted.

[0079] [Flowchart for object detection processing of still images]

[0080] based on Figure 7 illustrate Figure 6 Detailed information on object detection processing in S200. Figure 7This is a flowchart illustrating an example of object detection processing for a still image. Object detection processing is a preliminary stage in the identity determination method.

[0081] In S220, the object detection unit 101 performs initialization processing. The following is based on... Figure 8 A specific example of the initialization process is explained. Next, in S230, the object detection unit 101 obtains the next still image, which is the object to be detected, from the moving image file. Furthermore, if the initialization process has just occurred, the object detection unit 101 obtains the first still image from the moving image.

[0082] In step S240 (object detection step), the object detection unit 101 performs object detection on the still images obtained in step S230 and saves the detection results. The saving destination is, for example, the high-speed storage unit 11. Furthermore, in step S250, the object detection unit 101 determines whether to end the object detection process. Specifically, if the object detection unit 101 has performed object detection on all still images extracted from the moving images, it determines that the object detection process is to end ("Yes" in S250); otherwise, if there are still images for which object detection has not yet been performed, it determines that the process is not to end ("No" in S250). If the determination is "Yes" in S250, the process in step S200 ends; if the determination is "No" in S250, the process returns to step S230. If the process in step S200 ends, the process in step S300, i.e., node creation, is performed.

[0083] [Example of data structure for storing object detection results for still images]

[0084] In S240, for example, can be used Figure 8 The data structure shown stores the object detection results. Figure 8 This is a diagram illustrating an example of a data structure used to store object detection results for still images. Figure 8 The data structure includes an array of detection arrays 110 and an array of detection quantities 111. Additionally, this data structure also includes information 112 representing the number of images in the moving image and information 113 representing the number of classifications.

[0085] The detection array 110 is an array of three-dimensional data structures (image number, identification information of detected objects, detection number). The detection quantity 111 is a two-dimensional data structure representing the number of detections within the identification information of the detection array 110.

[0086] The aforementioned image numbers are identification numbers extracted from still images in a moving image, and can be arranged according to, for example, in... Figure 7 The sequence setting image number obtained in S230. Figure 8In this example, five values ​​from 0 to 4 are recorded as identification information for the detected object. These values ​​represent the classification of the detected object. For example, the identification information 0 to 4 could represent the classification of the detected object as corrugated cardboard, board, wood, mat, and long object, respectively. In this case, the object detection unit 101 uses a learned model obtained through machine learning to detect objects that fit into each of these classifications. Furthermore, corrugated cardboard is not an inappropriate object in these classifications, but it is included in the classification to distinguish it from board (to prevent false detection of board that is similar to corrugated cardboard).

[0087] The detection array 110 contains information representing the rotation of the detected object, the coordinates of its center position, its size (horizontal and vertical widths), and its probability. The rotation, center position, and size are calculated based on information provided during learning. The rotation of the detected object can be represented, for example, by the angle of rotation relative to a reference axis. The unit of the angle can be degrees (°) or radians. Furthermore, the probability is a numerical value representing the certainty of the detection result. This probability can be a value between 0 and 1, indicating the degree of probability with which the detected object can represent a classified object based on its image.

[0088] The center position can be represented using pixels in the image. Alternatively, it can be represented by a value such as (pixels in the image) / (pixel width of the image). In this case, the center position coordinates are in the range of 0 to 1. Furthermore, the center position can also be represented by a value obtained by converting the pixel values ​​to an actual size such as mm using methods such as projection transformation. Additionally, consider cases where the same object is detected multiple times (e.g., objects detected as different categories).

[0089] In application Figure 8 In the case of data structures, Figure 7 In the initialization process of S220, the object detection unit 101, for example, sets the detection array 110 and the detection quantity 111 to empty, sets the value of information 112 indicating the number of images to 0, and sets the value of information 113 indicating the number of classifications. For example, if the identification information of the detected object is 5 (0 to 4), then the value of information 113 is 5. Furthermore, the classification information used can be obtained from, for example, a file stored in the mass storage unit 12. Additionally, if the number of still images obtained from the moving images can be confirmed in advance, the area corresponding to all still images can be secured in S220.

[0090] Furthermore, in S240, the object detection unit 101 updates the detection array 110 and the number of detections 111 based on the information detected by object detection. Specifically, the object detection unit 101 adds regions of still images obtained in the preceding S230 processing to the detection array 110 and the number of detections 111. In the detection array 110, a region of the 112th image number (e.g., if the number of images 112 is 3, then the third one starting from 0) is added to the first dimension, a region of the 113th category number is added to the second dimension, and a data structure of the number of detections for each category is added to the third dimension. Furthermore, the number of images and the number of categories are referenced... Figure 8 The information of 112 and 113 is determined. Furthermore, the object detection unit 101 updates the information of each detected object to the area added to the detection array 110.

[0091] Regarding the detection quantity 111, the object detection unit 101 adds the image quantity 112 regions to the first dimension and the classification quantity 113 to the second dimension. Furthermore, the object detection unit 101 updates the number of objects in each detected classification. Finally, the object detection unit 101 increments the information 112 representing the number of images by 1.

[0092] [Example of a root graph]

[0093] Figure 9 This is a diagram illustrating an example of a root diagram created through the processes S300 and S600. The root diagram is a graph of the structure obtained by connecting nodes with branches, created according to each category of the detected object. Figure 9 The example shows curves 0 and 0 of object A. Figure 2 And an example of the curve of object B is shown. Figure 1 .

[0094] The root graph is created through two stages of processing: S300, which involves creating nodes, and S600, which involves connecting nodes corresponding to the same object in the created nodes using branches. Objects detected in each of multiple frame images are designated as nodes. Figure 9 In the example, the still images, i.e., frame images, extracted from the moving images are arranged from top to bottom in ascending order of their image numbers. Additionally, the nodes are numbered sequentially from 0 to 24, starting from the top. When using... Figure 5 In the case of a still image extracted from a dynamic image showing trash sliding down an inclined plane 600, as shown, the nodes forming a curve are positioned higher up, indicating detection on the upstream side of the inclined plane 600. Furthermore, it can be compared with... Figure 9 The reverse order, i.e., the images are arranged in descending order of image number, in which case the lower side of the curve corresponds to the upper side of the slope 600.

[0095] In a graph, the upstream node is called the root. Additionally, the node upstream of a given node is called the parent node, and the node downstream is called the child node.

[0096] exist Figure 9 In the example, branches are used to connect nodes corresponding to the same object in images whose image numbers differ by 2, and branches are also used to connect nodes corresponding to the same object in images whose image numbers differ by 1. In other words, the line segment connecting the node corresponding to the object determined by the identity determination unit 104 to be the same object is a branch. For example, regarding node 0 of object A, branches are used to connect nodes 1 and 2 in image 1, and branches are used to connect node 4 in image 2.

[0097] The moving image object detection unit 105 uses a root map generated as described above to detect objects from a predetermined number or more consecutive frame images. Even if there are frame images where no object is detected, the moving image object detection unit 105 will detect the object if it is detected in frame images within a predetermined range before and after it.

[0098] Therefore, even if an object is not detected in a portion of the frame, it can still be accurately detected in a moving image. For example, Figure 5 While plate 620d shown in image 610 is difficult to detect, plates 620c and 620e in the preceding and following frames can be detected. Therefore, even if the object detection unit 101 cannot detect plate 620d, plate 620 can be accurately detected in the dynamic image composed of these frames. Furthermore, the above-mentioned range can be set according to the moving speed of the detected object, the frame rate of the dynamic image, and the time interval between frames.

[0099] In the generation of the root graph, it is possible to detect multiple child nodes from a single frame image for a given parent node. For example, in Figure 9 In the example, for the parent node 0, child nodes 1 and 2 are detected from image 1. In this case, one of child nodes 1 and 2 is a false detection.

[0100] In the root graph generated by the graph generation unit 102, the moving image object detection unit 105 can detect an object corresponding to a node as an object presented in the moving image if a predetermined number or more consecutive frames of images containing nodes connected by branches are detected. Since the probability of false detection lasting for more than several frames is low, this structure can reduce the possibility of falsely detecting objects from the moving image.

[0101] exist Figure 9In the example, the specified number is set to 5. In this case, the motion image object detection unit 105 detects objects in the root graph whose following relation holds true.

[0102] (Image number of the bottommost node) - (Image number of the root node) ≥ 4

[0103] The value on the right-hand side of the above equation can be set based on the moving speed of the detected object, the frame rate of the motion image, and the time interval between frames. When using the above equation to detect objects, the moving image object detection unit 105 only detects objects that are moving speed, frame rate, and frame interval. Figure 9 The curve in the root graph is 0.

[0104] [Example of a data structure for storing information related to the root graph]

[0105] Figure 10 This is a graph that represents an example of a data structure used to store information related to the root graph. Figure 10 The various data shown are stored, for example, in high-speed storage unit 11. Figure 10 The data structure includes a node array 115 and information 116 indicating the number of nodes. Furthermore, the node array 115 is a structure that identifies and manages each node using a node number assigned to it. Moreover, for each node, the parent node number is associated with information determining the detection array of that node, an array of information determining the detection array of child nodes, and information representing the array of child node numbers.

[0106] Using this data structure, it is possible to represent each created node, its parent node, and its child nodes. That is, according to... Figure 10 The data structure is capable of representing Figure 9 That's the structure of the root graph. Of course, the data structure representing the root graph is arbitrary and not limited to... Figure 10 Examples.

[0107] [Node creation and processing flow]

[0108] based on Figure 11 Explain the process of node creation and processing. Figure 11 This is a flowchart illustrating an example of node creation processing (identity determination method). Furthermore, for those employing... Figure 10 The processing of data structures will be explained.

[0109] In S320, the graph generation unit 102 performs initialization processing. For example, as initialization processing, the graph generation unit 102 may perform... Figure 10 The node array 115 in the data structure is set to empty, the value of the information 116 representing the number of nodes is set to 0, and the current image number of the image number being processed is set to 1.

[0110] In S330, the graph generation unit 102 determines the image number of the old image, i.e., the old image number. The following will explain in detail how the identity of an object detected in the image with the current image number is determined with that of an object detected in the image with the old image number, and the node corresponding to the object determined to be the same object is determined as a structural element of a root graph.

[0111] exist Figure 11 In the example, the old image number is obtained by subtracting 2 from the current image number. In this case, the frames extracted from the moving image that differ from the current image number by 2 are the objects for determining identity. For example, based on... Figure 9 As explained, the scope of the identity determination is arbitrary, and the value subtracted from the current image number in S330 is also arbitrary. Furthermore, if the value subtracted from the current image number is negative, the old image number is set to 0.

[0112] In S340, the graph generation unit 102 sets the identification number (hereinafter referred to as the classification identification number) of the category being processed to 0. Next, in S350, the graph generation unit 102 obtains the detection results of the current image and the old image, and sets the current detection number to 0. More specifically, the graph generation unit 102 obtains... Figure 6 The object detection results in S200 are from the current image and old images, specifically the detection results of objects with a classification number of 0. Furthermore, these detection results can be obtained from, for example... Figure 8 It can be obtained from the detection array 110 as shown.

[0113] In S360, the graph generation unit 102 determines whether there are any unprocessed current detection results. Specifically, the graph generation unit 102 determines whether the current detection number is less than the total number of all current detection results obtained in S350. If this determination result is true, since there are unprocessed current detection results (that is, the determination result of S360 is "yes"), the process moves to S370. On the other hand, if the result is false, the determination result of S360 is no, and the process moves to S440.

[0114] In S370, the curve graph generation unit 102 obtains the current detection result using the current detection number and sets the old detection number to 0. Also, in S380, the curve graph generation unit 102 determines whether there are any unprocessed old detection results. Specifically, the curve graph generation unit 102 determines whether the old detection number is less than the number of old current detection results obtained in S350. If the determination result is true, since there are unprocessed old detection results (that is, the determination result of S380 is "yes"), the process moves to S390. On the other hand, if the result is false, the determination result of S380 is no, and the process moves to the processing of S430.

[0115] In S390, the identity determination unit 104 confirms the conditions for determining whether the detected object represented by the detection result for the current detection number is the same object as the detected object represented by the detection result for the old detection number. This condition can be related to, for example, the size, rotation angle, and position of the detected object. For example, the identity determination unit 104 can calculate the following values in S390.

[0116] Angle difference = Rotation angle of old detection - Rotation angle of current detection

[0117] X difference = X coordinate value of old detection - X coordinate value of current detection

[0118] Y difference = Y coordinate value of old detection - Y coordinate value of current detection

[0119] Horizontal width ratio = Horizontal width of old detection / Horizontal width of current detection

[0120] Vertical width ratio = Vertical width of old detection / Vertical width of current detection

[0121] Also, the identity determination unit 104 can determine identity based on whether the above values satisfy the following conditions. In addition, the inequality signs in each of the following conditions can be replaced with inequality signs with equality.

[0122] Angle condition: Absolute value of angle difference < Angle threshold

[0123] X condition: Lower limit value of X threshold < X difference < Upper limit value of X threshold

[0124] Y condition: Absolute value of Y difference < Y threshold

[0125] Horizontal width condition: Lower limit value of horizontal width ratio < Horizontal width ratio < Upper limit value of horizontal width ratio

[0126] Vertical width condition: Lower limit value of vertical width ratio < Vertical width ratio < Upper limit value of vertical width ratio

[0127] As Figure 5As shown, the closer to the downstream side of the 60° slope (the negative direction of the X-axis in the image), the smaller the object appears. Therefore, for images with larger image numbers, the upper and lower limits of the aforementioned horizontal and vertical widths can be set to smaller values. This reduces the possibility of false detections.

[0128] Furthermore, there is no absolute value in condition X because, in this example, such as Figure 5 As shown, the object on the inclined plane 600 is set as the detection object. Figure 5 In image 610, the object moves from right (larger X value) to left (smaller X value). On the other hand, since the object on the inclined plane 600 can move in either a positive (upward) or negative (downward) direction along the Y-axis, there is an absolute value in the Y condition. Thus, when there is a constraint on the direction of object movement in the image, by setting conditions corresponding to that constraint, the possibility of false detections can be reduced.

[0129] As described above, when the object detection unit 101 detects the position information (X value and Y value in the example above) of two objects that are the objects of identity determination, the identity determination unit 104 determines that the two objects are the same object based on the condition that these position information are in a predetermined relationship. As such, the possibility of false detection can be reduced.

[0130] Furthermore, the aforementioned thresholds can be adjusted based on factors such as the frame rate (fps) of the moving image file or the speed of the object moving on the inclined plane 600. For example, the thresholds can be set to values ​​proportional to the difference in image numbers (the value obtained by subtracting the old image number from the current image number). For instance, the value obtained by multiplying 10° by the difference in image numbers can be used as the angle threshold. Additionally, the X threshold (lower limit, upper limit), Y threshold, horizontal width threshold (lower limit, upper limit), and vertical width threshold (lower limit, upper limit) also depend on the unit used (e.g., pixels, or converted mm).

[0131] Furthermore, the object detection unit 101 can detect both the rotation angle of the object and the sine (angle) of the rotation angle. In this case, the angle can be calculated based on the detected rotation angle value and the aforementioned angle condition can be determined, or the angle condition can be changed to allow determination using the sine value of the rotation angle. Similarly, the cosine or tangent of the rotation angle can be used, or a combination thereof can be used. For example, if the object detection unit 101 detects the rotation angle as a combination of sine (angle) and cosine (angle), whether the angle condition is met can be confirmed by considering both arcsine (detection information) and arccosine (detection information) or by averaging them.

[0132] In S392, the replacement unit 103 determines whether to perform a replacement of the longitudinal width and the transverse width. For example, the replacement unit 103 can make this determination based on whether the absolute value of the angle difference is above a predetermined threshold (hereinafter referred to as the replacement threshold). In this case, if the absolute value of the angle difference is above the replacement threshold, the replacement unit 103 determines to perform a replacement ("Yes" in S392); if the absolute value of the angle difference is less than the replacement threshold, the replacement unit 103 determines not to perform a replacement ("No" in S392). The desired replacement is based on... Figure 16 The replacement is performed when the detected value of the rotation angle deviates by 90°. Therefore, the replacement threshold can be set to, for example, 60°. In S392, if it is determined that a replacement should be performed, the process moves to S394; if it is determined that no replacement should be performed, the process moves to S400.

[0133] Thus, when the difference in rotation angles between two objects that are the objects of identity determination is above a predetermined threshold (replacement threshold), the replacement unit 103 replaces the longitudinal width and lateral width of one of the objects. Therefore, from the viewpoint of the difference in rotation angles, efficient processing can be achieved when the orientation of the axis used as the reference for calculating the rotation angle is likely to differ by 90°. Furthermore, if efficiency is not sought, the identity determination unit 104 can be structured such that identity determination is performed twice, before and after replacement by the replacement unit 103, and if either determination result is the same object, then it is determined to be the same object.

[0134] In step S394 (replacement step), the replacement unit 103 replaces the longitudinal width and the transverse width. Additionally, the identity determination unit 104 adjusts the absolute value of the aforementioned angle difference. Specifically, the replacement unit 103 replaces the longitudinal width and transverse width for either the object represented by the detection result of the current detection number or the object represented by the detection result of the old detection number. Furthermore, the replacement unit 103 uses the replaced longitudinal width and transverse width values ​​to enable the identity determination unit 104 to determine whether the transverse width condition and the longitudinal width condition are satisfied. Additionally, the identity determination unit 104 uses the value obtained by subtracting 90° from the absolute value of the aforementioned angle difference to determine whether the angle condition is satisfied.

[0135] Thus, when the object detection unit 101 detects the rotation angle of an object, the identity determination unit 104 determines whether they are the same object based on the value after correcting the difference in the object's rotation angle by 90° and the values ​​of the longitudinal width and transverse width replaced by the replacement unit 103. Therefore, even if the reference axis is offset by 90° when detecting the rotation angle of two objects that are the objects of identity determination, the identity of the objects can be accurately determined by appropriately taking into account the actual difference in rotation angle.

[0136] For example, based on Figure 16 When determining identity based on the detection results of the learned model b, the replacement unit 103 can replace the vertical and horizontal widths of the box 1120b. In this case, the identity determination unit 104 determines whether the above conditions are met based on the vertical width (32) and horizontal width (10) of the box 1020b, and the vertical width (32) and horizontal width (10) of the replaced box 1120b. In addition, the identity determination unit 104 determines whether the above conditions are met based on the value obtained by subtracting 90° from the absolute value of the difference between the angle (-9°) of the box 1020b and the angle (97°) of the box 1120b, which is 16°. Thus, it is possible to accurately determine that the object in the box 1020b and the object in the box 1120b are the same object.

[0137] Furthermore, when the object detection unit 101 outputs sine (angle) and cosine (angle) as information representing the rotation angle of the detected object, the replacement unit 103 only needs to replace the output sine (angle) and cosine (angle). Alternatively, instead of correcting the difference in the rotation angles of the objects by 90°, the difference in the rotation angles of the two objects that are the objects for determining identity can be calculated after correcting the rotation angle of one of them by 90°. Furthermore, it is also possible to determine the identity of objects without considering the rotation angle.

[0138] In step S400 (identity determination step), the identity determination unit 104 determines whether the object represented by the detection result of the current detection number is the same object as the object represented by the detection result of the old detection number. For example, the identity determination unit 104 may determine that the object is the same object ("yes" in S400) if all conditions (angle, X, Y, horizontal width, and vertical width) of S390 are true, and determine that the object is not the same object ("no" in S400) if any condition is not true. Furthermore, if an adjustment has been made in S394, the identity determination unit 104 uses the adjusted value to determine whether the conditions of S390 are met. If the determination is yes in S400, the process moves to S410; if the determination is no in S400, the process moves to S420.

[0139] Furthermore, the identity determination unit 104 can only verify a portion of the conditions in S390. For example, when the identity determination unit 104 determines the identity of a non-rotating object, the angle condition does not need to be considered. In addition, when determining the identity of an object that does not move in the X direction, the X condition does not need to be verified, and when determining the identity of an object that does not move in the Y direction, the Y condition does not need to be verified.

[0140] In S410, the graph generation unit 102 updates the nodes of the root graph. For example, the graph generation unit 102 may (1) detect or add information append nodes as append information nodes based on the determination result of S400, and (2) update the nodes of the root graph using the two steps of adding information to the information append nodes.

[0141] In this case, the graph generation unit 102 checks all nodes in the node array 115 to see if any old detection exists. Furthermore, the total number of nodes is indicated in the node count information 116. That is, the graph generation unit 102 determines that an old detection exists if any of the following conditions are met for each node.

[0142] The image number shown in node array 115 is the same as the old image number.

[0143] The classification and identification of the detected object shown in node array 115 are consistent with the classification and identification number.

[0144] The detection number in the identification information shown in node array 115 is consistent with the old detection number.

[0145] Furthermore, the graph generation unit 102 uses nodes that have been previously detected, i.e., nodes that are determined to meet any one of the above three conditions, as information appending nodes. In addition, in one processing step S410, the information described below is appended to an information appending node, so after any node in the node array 115 is used as an information appending node, it is not necessary to confirm the remaining nodes.

[0146] Even if the graph generation unit 102 performs the above-described verification on all nodes in the node array 115, if no previously detected nodes are found, the graph generation unit 102 will add new nodes to the node array 115 and treat the added nodes as information addition nodes. Furthermore, the graph generation unit 102 increments the value of the information 116 indicating the number of nodes by 1. Moreover, the graph generation unit 102 adds the following information to the information addition nodes added to the node array 115.

[0147] Image number in moving images = old image number

[0148] The classification and identification of the detected item = classification and identification number

[0149] The detection number in the identification information = the old detection number

[0150] Furthermore, all items in the arrays of parent node numbers and child node numbers are set to undefined. That is, all items in these arrays are undefined. Furthermore, undefined can be represented by a value such as "-1".

[0151] Furthermore, for information addition nodes that are detected or added as described above, the graph generation unit 102 adds the following information to the undefined items in the determination information of the detection array of child nodes in the node array 115.

[0152] Image number = current image number

[0153] The classification and identification of the detected item = classification and identification number

[0154] The detection number in the identification information = the current detection number

[0155] In S420, the graph generation unit 102 increments the old detection number by 1, and then returns to the processing in S380. In S430, the graph generation unit 102 increments the current detection number by 1, and then returns to the processing in S360. In S440, the graph generation unit 102 increments the classification identification number by 1, and then the processing moves to S450.

[0156] In S450, the graph generation unit 102 determines whether the determination of all categories is complete. Specifically, the graph generation unit 102 compares the category identification number with the number of categories (in... Figure 8 If the number of categories represented by information 113 is not equal (in S450 it is "No"), the process returns to S350; if they are equal (in S450 it is "Yes"), the process proceeds to S460.

[0157] In S460, the graph generation unit 102 increments the old image number by 1, and then the process moves to S470. In S470, the graph generation unit 102 determines whether the determination of all old images is complete. Specifically, the graph generation unit 102 compares the old image number with the current image number. If they are not equal ("No" in S470), it returns to the process in S340; if they are equal ("Yes" in S470), it proceeds to the process in S480.

[0158] In S480, the graph generation unit 102 increments the current image number by 1, and then the process moves to S490. In S490, the graph generation unit 102 determines whether the determination of using all images as the current image has ended. Specifically, the graph generation unit 102 compares the current image number with the number of images (in... Figure 8If the number of images (represented by information 112) is not equal (in S490, this is "No"), the process returns to S330; if they are equal (in S490, this is "Yes"), the process ends. Figure 11 The processing. In Figure 11 After the processing is completed, proceed with Figure 6 Node connection processing of S600.

[0159] [Node connection processing flow]

[0160] based on Figure 12 Explain the process of node connection handling. Figure 12 This is a flowchart illustrating an example of node connection processing. In the node connection process, the array of parent and child node numbers in node array 115, which were not set in the node creation process described above, is updated. This update is equivalent to connecting nodes with branches.

[0161] exist Figure 12 In the example, three appended variables are used: the current node number, the old node number, and the child node array number. These appended variables are stored, for example, in high-speed storage unit 11.

[0162] In S620, the graph generation unit 102 performs initialization processing. For example, in the initialization processing, the graph generation unit 102 sets the value obtained by subtracting 1 from the number of nodes as the current node number. Furthermore, as... Figure 10 As shown, the number of nodes can be determined based on the information 116 representing the number of nodes. Furthermore, in S630, the graph generation unit 102 sets the value obtained by subtracting 1 from the current node number as the old node number. Moreover, in S640, the graph generation unit 102 sets the child node array number to 0.

[0163] In S650, the graph generation unit 102 performs parent-child node verification, that is, verifies whether the child nodes of the old node are consistent with the current node. Specifically, the graph generation unit 102 determines whether the determination information of the detection array of the child nodes in the old node number of the node array 115 is consistent with the determination information of the detection array of the child nodes in the current node number of the node array 115. Furthermore, the determination information of the detection array of the child nodes includes the image number, the classification and identification of the detected object, and the detection number in the identification information. If all these are consistent, the determination result of S650 is yes, and the process moves to S660. On the other hand, if there is inconsistent information, the determination result of S650 is no, and the process moves to S670.

[0164] In S660, the graph generation unit 102 updates the parent and child nodes, that is, it updates the array of parent node numbers and child node numbers. Specifically, the graph generation unit 102 sets the first unset item in the array of child node numbers of the old node number in the node array 115 to the current node number. In addition, the graph generation unit 102 sets the parent node number of the current node number in the node array 115 to the old node number.

[0165] In S670, the graph generation unit 102 increments the child node array number by 1. Furthermore, in S680, the graph generation unit 102 determines whether the processing of the child nodes is complete. Specifically, the graph generation unit 102 determines whether the child node array number is smaller than the number of items in the array of determination information for the child node detection array in the node array 115. If the determination is smaller ("No" in S680), the graph generation unit 102 returns to the processing in S650; otherwise ("Yes" in S680), it proceeds to the processing in S690.

[0166] In S690, the graph generation unit 102 decrements the old node number by 1. Furthermore, in S700, the graph generation unit 102 determines whether the processing of the old node is complete. Specifically, the graph generation unit 102 determines whether the old node number is a positive value. If the graph generation unit 102 determines that it is a positive value ("No" in S690), it returns to the processing in S640; otherwise ("Yes" in S690), it proceeds to the processing in S710.

[0167] In S710, the graph generation unit 102 decrements the current node number by 1. Furthermore, in S720, the graph generation unit 102 determines whether the processing of the current node is complete. Specifically, the graph generation unit 102 determines whether the current node number is a positive value. If the current node number is determined to be a positive value ("No" in S720), the graph generation unit 102 returns to the processing in S630; otherwise ("Yes" in S720), the process ends. Figure 12 The processing. In Figure 12 After the processing is completed, proceed with Figure 6 Object detection processing for dynamic images in the S800. Furthermore, in... Figure 12 In this process, the current node number and the old node number are reduced from a large number to a smaller number (by 1 each time). It can also be the opposite process (setting the initial value of the current node number and the old node number to 0 and incrementing by 1 each time).

[0168] [Flowchart for object detection processing of dynamic images]

[0169] based on Figure 13 This section describes the process of object detection and processing for dynamic images. Figure 13This is a flowchart illustrating an example of object detection processing for a moving image. In the object detection processing of the moving image, a root graph, constructed as described above, is used; specifically, a completed node array 115 is used to detect objects. In this processing, objects are detected based on the length of the root graph, i.e., the number of images from the root of the root graph to the latest node.

[0170] In S820, the moving image object detection unit 105 performs an initialization process. For example, in the initialization process, the moving image object detection unit 105 sets the old node number to 0. Next, in S830, the moving image object detection unit 105 determines whether the node with the old node number is the root. Specifically, if the parent node number of the old node number in the node array 115 is not set, the moving image object detection unit 105 determines it to be yes in S830 and proceeds to the process in S840; if it has been set, it determines it to be no in S830 and proceeds to the process in S880.

[0171] In S840, the motion image object detection unit 105 explores the latest node in the root graph.

[0172] This refers to the node of the frame image with the largest image number among a series of nodes connected by branches. Specifically, the moving image object detection unit 105 recursively (details omitted) traces all nodes of the root graph starting from the oldest node number to explore the latest node. Furthermore, the moving image object detection unit 105 sets the node number of the latest detected node as the current node number.

[0173] In S850, the moving image object detection unit 105 calculates the difference between the image number of the determination information of the detection array set at the current node number node in the node array 115 and the image number of the determination information of the detection array set at the old node number node in the node array 115. Furthermore, in S860, the moving image object detection unit 105 determines whether the difference calculated in S850 is 4 or more. If the result is 4 or more (yes in S860), it moves to S870 as if an object has been detected; if it is less than 4 (no in S860), it moves to S880. Furthermore, based on... Figure 9 As explained, the value "4" used as the criterion can be adjusted according to the situation.

[0174] In S870, the moving image object detection unit 105 stores information related to the detected object. For example, the moving image object detection unit 105 can store the frame image corresponding to each node of the root map used for object detection and the object information of the object in association. Figure 8 The information described in the detection array 110 shown is used as object information.

[0175] In S880, the moving image object detection unit 105 increments the old node number by 1. Furthermore, in S890, the moving image object detection unit 105 determines whether the processing of all old nodes is complete. Specifically, the moving image object detection unit 105 determines whether the old node number is smaller than the number of nodes (the number of nodes determined from the information 116 representing the number of nodes). If the determination is smaller (yes in S890), the moving image object detection unit 105 returns to the processing in S830; otherwise (no in S890), it terminates the process. Figure 13 The processing involves multiple arrays. Furthermore, the example above illustrates the use of multiple arrays, but linked lists can also be used instead.

[0176] Furthermore, while object detection was performed according to classification in this example, there are objects that are difficult for even humans to distinguish, such as boards and wood. Human judgment errors may also occur when creating such teaching data. Therefore, when detecting objects, the boundary between boards and wood is ambiguous; the same object may sometimes be classified as a board and sometimes as wood based on the image. To improve this problem, the classification can be grouped and then used for same-object detection. This could involve treating boards and wood as one group, or grouping all inappropriate objects into a single group.

[0177] For example, the graph generation unit 102 can create a root graph by using all detected objects belonging to the same group as nodes. A group only needs to contain objects that might be falsely detected, such as boards and corrugated paper. When using such a root graph to detect objects, the moving image object detection unit 105 can detect a certain object as an object displayed in the moving image, even if a predetermined number or more consecutive frames contain frames where no object was detected, when an object belonging to the same group as the object is detected in a frame where the object was never detected. Therefore, even if false detection of an object occurs in a portion of consecutive frames, the object can be accurately detected.

[0178] [Processing for visual verification]

[0179] based on Figure 14 The process of visually confirming whether an inappropriate object is present in the image detected by the information processing device 1 using the selection display device 4 is explained. Figure 14 This diagram illustrates the procedures used for visual verification. In Figure 14 The diagram shows a flowchart illustrating an example of a visual confirmation process, and an example of a screen displayed on the selection display device 4 using the process described in the flowchart.

[0180] In S1010, the display control unit 106 is selected for initialization processing. During the initialization process, the display control unit 106 is selected to obtain... Figure 6 A list of moving image files obtained by detecting inappropriate objects through the processing of Figure 6 is created, and the number of the moving image file to be processed is set to zero. Additionally, it is not necessary to obtain a list of all moving image files that have undergone the processing of Figure 6 . For example, the selection display control unit 106 can utilize time periods, file name reduction (Japanese: 絞り込む) to narrow down the moving image files included in the list, or can obtain a list of moving image files stored after the end time of the most recently performed processing of Figure 6 . Figure 6 A list of all moving image files that have undergone the processing of Figure 6 is not required. For example, the selection display control unit 106 can utilize time periods, file name reduction (Japanese: 絞り込む) to narrow down the moving image files included in the list, or can obtain a list of moving image files stored after the end time of the most recently performed processing of Figure 6 . Figure 14 A list of moving image files stored after the end time of the most recently performed processing of Figure 14 .

[0181] In S1020, the selection display control unit 106 determines whether there are still files in the moving image files included in the moving image list that have not undergone the processing after S1030. Specifically, the selection display control unit 106 determines the truth or falsehood of "the number of the moving image file to be processed < the total number of all files included in the moving image file list". Moreover, when the determination result is true (in S1020, "yes"), the selection display control unit 106 proceeds to the processing of S1030, and when the determination result is false (in S1020, "no"), it proceeds to the processing of S1050.

[0182] In S1030, the selection display control unit 106 selects the images to be displayed on the selection display device 4 from the moving image files to be processed. The selection method is not particularly limited. For example, the selection display control unit 106 can select all the images in the root diagrams created through the processing of Figure 6 from the moving image files to be processed, or can also select a part of the images. When selecting a part of the images, the upper limit number of images set for each moving image file can be selected, or the upper limit number of images set for each root diagram can be selected. Moreover, in S1040, the selection display control unit 106 increments the number of the moving image file to be processed by 1. After that, the process returns to S1020. Figure 6 A list of all the images in the root diagrams created through the processing of Figure 6 , or a part of the images can also be selected. When selecting a part of the images, the upper limit number of images set for each moving image file can be selected, or the upper limit number of images set for each root diagram can be selected. Moreover, in S1040, the selection display control unit 106 increments the number of the moving image file to be processed by 1. After that, the process returns to S1020.

[0183] Furthermore, when selecting a subset of images, the selection can be random or prioritized for images with higher probability values ​​output by the object detection unit 101. When the object detection unit 101 outputs a probability value, representing the certainty of the detection result, the selection display control unit 106 can display frame images selected based on probability values ​​from a predetermined number or more consecutive frame images on the display device. For example, when selecting two images from a root image, the selection display control unit 106 can determine the probability output by the object detection unit 101 for each node in the root image and select the image corresponding to the node with the highest probability and the image corresponding to the node with the second highest probability. Since there is a tendency for the probability value to be higher the more clearly an object is presented in a frame image, according to the above structure, frame images that are easy for the person in charge of visual inspection to confirm the appropriateness can be displayed.

[0184] Furthermore, when the selection display control unit 106 selects multiple images, a time condition between the selected images can be added. For example, the selection display control unit 106 can select images with a time difference of 5 seconds or more. Alternatively, the selection display control unit 106 can select images in order of large time differences (e.g., a maximum of N). In this case, the selection display control unit 106 does not select from all the images constituting the root graph, but can select from a subset of images (e.g., images randomly selected from the root graph or selected based on probability values, etc.). Alternatively, the selection display control unit 106 can select N images from all the images constituting the root graph in order of large time differences, and then select an image from those N images based on probability values, etc.

[0185] Furthermore, when the motion image object detection unit 105 detects an object at multiple locations in the motion image, the selection display control unit 106 can display frame images of locations with an interval of more than a predetermined time on the selection display device 4. This reduces the likelihood of displaying frame images showing the same object and allows for efficient visual confirmation. Furthermore, detecting an object at multiple locations in the motion image refers to the motion image object detection unit 105 detecting an object from multiple root maps created based on the motion image file. In such cases, if the distance between two root maps is close, or more precisely, if the interval from the latest node of the upstream root map to the root of the downstream root map is short, these root maps may correspond to the same object. Therefore, by adopting the above structure, the likelihood of displaying frame images showing the same object can be reduced. Moreover, the interval between root maps can be expressed in terms of the number of images or time.

[0186] Furthermore, the object detection unit 101 of this embodiment detects objects belonging to multiple categories. When the moving image object detection unit 105 detects objects of different categories at multiple positions in the dynamic image, the selection display control unit 106 can display the frame image at each position on the selection display device 4. On the other hand, when the moving image object detection unit 105 detects objects of the same category at multiple positions in the dynamic image, the selection display control unit 106 can display the frame image at any one position on the selection display device 4. Thus, visual confirmation can be performed separately for objects of different categories, and visual confirmation can be completed at once for objects of the same category, thereby enabling efficient visual confirmation.

[0187] In S1050, the selection display control unit 106 sends the image selected in S1030 to the selection display device 4 via the selection display IF unit 15 and displays it. Furthermore, when displaying the image, the selection display control unit 106 can perform any one or more of the following processes (1) to (7): (1) Arrange and display the images according to the motion picture file. (2) Also display the date and time of the motion picture file. (3) Display information related to the selection of the image (e.g., checkboxes) associated with the image. (4) Highlight the detected object (e.g., surround the object with lines at the four corners and display it). (5) Also display information associated with the object (e.g., the probability output by the object detection unit 101, the vehicle to be transported determined by the vehicle to be transported unit 107, etc.). (6) Cut out and display a portion of the detected object from the image. (7) Enlarge the display.

[0188] exist Figure 14 In the example, the selection display device 4 displays images of objects (specifically boards and long objects) detected as inappropriate objects from motion picture file XX captured on 2019 / xx / xx. Additionally, the selection display device 4 displays images of objects (specifically garbage bags and corrugated cardboard) detected as inappropriate objects from motion picture file YY captured on 2019 / yy / yy. Furthermore, in each object's image, a frame surrounding the detected object is displayed, and a checkbox is also displayed.

[0189] In S1060, the selection of the image by the person in charge of visual confirmation is handled by the display control unit 106. For example... Figure 14 As shown in the example, when a checkbox is displayed, the selection control unit 106 determines that the image corresponding to the checked checkbox is the selected image.

[0190] An image of the object is acceptable. Furthermore, the selection display control unit 106 displays the end button along with the image, as described above. When this end button is selected, it can be determined that the image selection has ended. Moreover, in S1070, the selection display control unit 106 saves the selected image in the high-speed storage unit 11 or the large-capacity storage unit 12, thereby ending the selection process. Figure 14 The processing.

[0191] [Processing of images displaying inappropriate objects]

[0192] based on Figure 15 This describes the process of displaying an image of an object on the object display device 5. Figure 15 This diagram illustrates the processing of images that display inappropriate objects. Figure 15 The diagram shows: a flowchart illustrating an example of the process for displaying an image of an inappropriate object, and an example of a screen displayed on the inappropriate object display device 5 using the process described in the flowchart. Furthermore, as a prerequisite for this process, it is assumed that the vehicle information collection device 3 obtains identification information of a garbage collection vehicle 200 that is transporting garbage to the waste incineration facility and sends it to the information processing device 1.

[0193] In S1110, the vehicle identification unit 107 determines the garbage delivery vehicle based on the identification information received from the vehicle information collection device 3. For example, if the received identification information is an image of the garbage delivery vehicle, the vehicle identification unit 107 can determine the delivery vehicle by analyzing the image.

[0194] In step S1120, the improper object display control unit 108 determines whether an image of an improper object associated with the incoming vehicle identified in step S1110 exists. If the determination is yes in step S1120, the process proceeds to step S1130; if the determination is no in step S1120, the process ends. Figure 15 The processing.

[0195] Furthermore, as a prerequisite for making the determination in S1120, it is assumed that for each improper item detected by the information processing device 1, information indicating the vehicle transporting the improper item is stored in the large-capacity storage unit 12 or the like. Such a connection between the improper item and the transported vehicle can be established, for example, through... Figure 6 The processing is performed before and after. The vehicle identification unit 107 only needs to identify the vehicle being transported at the time of the association. Furthermore, if the vehicle identified in S1110 is one that previously transported improper items, the improper item display control unit 108 can also determine that it has an image of the associated improper item. Additionally, for the vehicle identified in S1110 that previously transported improper items... Figure 15 In the case of processing, the image stored after the previous processing can be used as an object for determination in S1120.

[0196] In S1130, the improper item display control unit 108 causes the improper item display device 5 to display an image of the improper item associated with the vehicle transported in, as determined in S1110. This concludes the process. Figure 15 The processing. Furthermore, the image displayed on the improper object display device 5 can end display at a time such as when the image of the garbage captured in S1110, indicating the garbage transported by the vehicle, ends. Figure 14 The processing time, the time elapsed after a specified period from the start of the display on S1130.

[0197] When an image is displayed in S1130, the object display control unit 108 may perform any one or more of the following processes (1) to (6): (1) If multiple motion image files of detected objects exist, the images are arranged and displayed according to the motion image files. (2) The date and time of the motion image files are also displayed. (3) The detected object is highlighted (e.g., surrounded by lines at the four corners and displayed). (4) Information associated with the object is also displayed (e.g., the probability output by the object detection unit 101). (5) A portion of the detected object is cropped from the image and displayed. (6) The image is enlarged for display.

[0198] exist Figure 15 In this example, the improper object display device 5 displays an image of an object (specifically a board and a long strip) detected as an improper object from a motion image file XX captured on 2019 / xx / xx. Additionally, the improper object display device 5 displays a frame surrounding the detected object and a message reminding users not to bring in improper objects.

[0199] [Variation Example]

[0200] In the object detection, object classification, and other tasks described above, artificial intelligence / machine learning algorithms other than neural networks that have undergone machine learning (including those with deep learning capabilities) can also be used.

[0201] The executing entities for each process described in the above embodiments can be appropriately changed. For example, they can be omitted. Figure 1 At least one of the boxes shown is used, and the omitted processing unit is provided in one or more other devices. In this case, the processing of each of the above embodiments is performed using one or more information processing devices.

[0202] Furthermore, while examples of detecting inappropriate objects from junk images have been described in the above embodiments, the detected object is arbitrary and not limited to inappropriate objects. The information processing device 1 can detect the object from each of multiple images of an arbitrary object captured in a time sequence, and can determine the identity of the detected object with high accuracy. For example, the information processing device 1 can also be used for tracking objects, people, etc.

[0203] This invention is not limited to the embodiments described above, and various modifications can be made within the scope of the claims. Embodiments obtained by appropriately combining technical solutions disclosed in different embodiments are also included within the technical scope of this invention.

Claims

1. An information processing device comprising: The object detection unit detects objects from each of a plurality of images captured in a time sequence, and detects the vertical width and horizontal width of the detected objects. The identity determination unit determines whether a first object detected from a first image among the plurality of images and a second object detected from a second image among the plurality of images are the same object; as well as The replacement part replaces the values ​​of the longitudinal width and the transverse width of the first object or the second object. When the replacement unit performs the replacement, the identity determination unit makes the determination based on the values ​​of the replaced vertical width and horizontal width. The multiple images are frame images extracted from dynamic images. The information processing device includes a moving image object detection unit. Regarding the motion image object detection unit, when objects detected in each of the plurality of frame images are designated as nodes, and line segments connecting to the nodes corresponding to objects determined by the identity determination unit to be the same object are designated as branches... If a predetermined number or more consecutive frame images of nodes connected by the branch are detected, the moving image object detection unit detects the object corresponding to the node connected by the branch as an object presented in the dynamic image. Even if a specified number or more of the frame images contain frame images in which the object was not detected, the moving image object detection unit will still detect the specified object as an object presented in the moving image if it detects the object in a frame image in which the object was never detected. The object is one that could be mistakenly detected as the specified object.

2. The information processing device according to claim 1, characterized in that, The object detection unit also detects the rotation angles of the first object and the second object. The identity determination unit determines whether the first object and the second object are the same object based on the value after correcting the difference between the rotation angles of the first object and the second object by 90°, or the difference between the rotation angles of the first object and the second object after correcting by 90°, and the values ​​of the longitudinal width and the transverse width replaced by the replacement unit.

3. The information processing device according to claim 2, characterized in that, When the difference in rotation angle between the first object and the second object is above a predetermined threshold, The replacement part replaces the vertical width and horizontal width of the first object. The identity determination unit determines whether the first object and the second object are the same object based on the value after correcting the difference in rotation angle between the first object and the second object by 90°, and the values ​​of the longitudinal width and transverse width replaced by the replacement unit.

4. The information processing apparatus according to any one of claims 1 to 3, characterized in that, The object detection unit also detects the position information of the first object and the second object. The identity determination unit determines that the first object and the second object are the same object based on the condition that the position information of the first object and the position information of the second object are in a predetermined relationship.

5. The information processing apparatus according to claim 1, characterized in that, Even if the frame images contain a specified number or more consecutive frames in which the object is not detected, the motion image object detection unit will detect the object as an object presented in the motion image if it detects the object in frame images within a specified range before and after the frame images in which the object was not detected.

6. The information processing apparatus according to claim 1 or 5, characterized in that, The information processing device includes a display control unit. When the moving image object detection unit detects an object at multiple locations in the dynamic image, the display control unit displays the frame image at a location with an interval of more than a predetermined time among the multiple locations on the display device.

7. The information processing apparatus according to claim 1 or 5, characterized in that, The object detection section detects objects belonging to multiple categories. The information processing device includes a display control unit. When the moving image object detection unit detects objects of different categories at multiple locations in the dynamic image, the display control unit causes the display device to display the frame image at each location. When the moving image object detection unit detects objects of the same category at multiple locations in the dynamic image, the display control unit causes the display device to display the frame image at any one location.

8. The information processing apparatus according to claim 1 or 5, characterized in that, The object detection unit outputs an index value representing the determinism of the detection result. The information processing device includes a display control unit, which causes the display device to display a frame image selected based on the index value from among a predetermined number or more consecutive frame images.

9. A method for determining identity, executed by one or more information processing devices, characterized in that, include: The object detection step detects objects from each of multiple images captured in a time sequence, and detects the vertical and horizontal widths of the detected objects. The replacement step involves replacing the vertical and horizontal width values ​​of a first object detected from a first image of the plurality of images or a second object detected from a second image of the plurality of images; and The identity determination step determines whether the first object and the second object are the same object based on the replaced vertical and horizontal width values. The multiple images are frame images extracted from dynamic images. The identity determination method also includes a moving image object detection step. In the moving image object detection step, when objects detected in each of the multiple frame images are taken as nodes, and line segments connecting to the nodes corresponding to objects determined by the identity determination unit to be the same object are taken as branches... If a predetermined number or more consecutive frame images of nodes connected by the branch are detected, the object corresponding to the node connected by the branch is detected as an object presented in the dynamic image. In the moving image object detection step, even if a specified number of consecutive frames contain frames in which the object was not detected, if a specified object is detected in a frame in which the object was never detected, that object will still be detected as an object presented in the moving image. The object is one that could be mistakenly detected as the specified object.

10. A computer-readable recording medium recording an identity determination program, the identity determination program being used to enable a computer to function as the information processing apparatus of claim 1, and to enable the computer to function as the object detection unit, the identity determination unit, the replacement unit, and the moving image object detection unit.

Images