Image processing apparatus, image processing method, and image processing program

Abstract

To provide an image processing device, an image processing method, and an image processing program capable of suppressing a processing load of object detection processing.SOLUTION: The object detection device includes an object position prediction unit that predicts a position of an object in a new input image based on a position of the object detected in a past input image, a second object detection unit that performs object detection on a partial region in the new input image, an object detection target region determination unit that determines an object detection target region in the new input image based on a prediction result by the object position prediction unit and an object detection result by the second object detection unit, and a first object detection unit that performs object detection on the object detection target region determined by the object detection target region determination unit.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] As a technique related to image processing, for example, Patent Document 1 discloses a technique related to object detection using a neural network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-036008 Summary of the Invention [Problem to be solved by the invention]

[0004] Object detection processing requires high throughput. For example, a system that performs object detection processing on images output from a surveillance camera needs to operate at a relatively high frame rate to prevent oversights (i.e., missed detections). However, if you try to achieve high throughput without reducing the processing load of object detection processing, problems arise such as an increase in the number of devices used for processing, increased costs, and increased power consumption.

[0005] The technology described in Patent Document 1 aims to improve the accuracy of detecting the position of an object by correcting information acquired during the estimation process based on object movement information acquired from a series of images. Therefore, the technology described in Patent Document 1 cannot be expected to reduce the processing load of the object detection process.

[0006] The present disclosure has been made in consideration of these problems, and an object of the present disclosure is to provide an image processing device, an image processing method, and an image processing program that can reduce the processing load of object detection processing. [Means for solving the problem]

[0007] The image processing device according to the present disclosure includes an object position prediction means that predicts the position of an object in a new input image based on the position of the object detected in a past input image; a second object detection means that performs object detection on a partial area in the new input image; an object detection target area determination means that determines an object detection target area to be the target of object detection in the new input image based on the prediction result by the object position prediction means and the object detection result by the second object detection means; and a first object detection means that performs object detection on the object detection target area determined by the object detection target area determination means.

[0008] In the image processing method disclosed herein, a computer predicts the position of an object in a new input image based on the position of the object detected in a past input image, performs object detection on a partial area in the new input image, determines an object detection target area to be the target of object detection in the new input image based on the prediction result and the object detection result, and performs object detection on the determined object detection target area.

[0009] The image processing program according to the present disclosure causes a computer to execute an object position prediction process that predicts the position of an object in a new input image based on the position of the object detected in a past input image; a second object detection process that performs object detection on a partial area in the new input image; an object detection target area determination process that determines an object detection target area to be the target of object detection in the new input image based on the prediction result of the object position prediction process and the object detection result of the second object detection process; and a first object detection process that performs object detection on the object detection target area determined in the object detection target area determination process. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to reduce the processing load of object detection processing. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram illustrating an example of a functional configuration of the image processing apparatus. [Figure 2] FIG. 10 is an explanatory diagram illustrating an outline of the operation of a second object detection unit. [Figure 3] 10 is a flowchart illustrating an example of an operation of the image processing device. [Figure 4] 10 is a flowchart illustrating an example of an operation of the image processing device. [Figure 5] FIG. 2 is a block diagram illustrating an example of a hardware configuration. [Figure 6] FIG. 2 is a block diagram illustrating an example of a functional configuration of the image processing apparatus. [Figure 7] 10 is a flowchart illustrating an example of an operation of the image processing device. [Figure 8] FIG. 1 is a block diagram illustrating a main part of an image processing device. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0013] Embodiment 1. [Configuration Description] The functional configuration of an image processing device according to a first embodiment will be described. FIG. 1 is a block diagram illustrating the functional configuration of an image processing device. The image processing device 1 shown in FIG. 1 includes an object position prediction unit 10, an object detection target area determination unit 20, a first object detection unit 30, a second object detection unit 40, an object position storage unit 50, an image input unit 60, and an object detection mode determination unit 70. Note that the number and connection relationships of the components shown in FIG. 1 are merely examples. For example, the image processing device 1 may be configured to include multiple image input units 60.

[0014] The image processing device 1 may be configured using a computer device including a CPU (Central Processing Unit), a main memory, and a secondary storage device. In this case, the object position prediction unit 10, object detection target area determination unit 20, first object detection unit 30, second object detection unit 40, object position storage unit 50, image input unit 60, and object detection mode determination unit 70 of the image processing device 1 shown in Fig. 1 are realized by the CPU executing processes in accordance with programs stored in the secondary storage device. The hardware configuration of the image processing device 1 will be described in further detail below.

[0015] In this embodiment, an example will be described in which the image processing device 1 uses an image as input and performs object detection of a person in the image. For example, the image processing device 1 executes an object detection task of a person class. However, the object detection tasks that the image processing device 1 can execute are not limited to the person class. Hereinafter, an image input to the image processing device 1 will also be referred to as an input image.

[0016] The object position prediction unit 10 predicts the position of an object in a newly input image based on the position of the object detected in a previously input image. Hereinafter, an object detected by object detection from an image is also referred to as a known object.

[0017] The first object detection unit 30 performs object detection of known objects in a newly input image. The second object detection unit 40 performs object detection in a newly input image to prevent new objects from being overlooked. A new object is, for example, an object whose past position information is not stored in the object position storage unit 50. A new object is, for example, an object whose past position information is not sufficiently stored in the object position storage unit 50 to be used as input for prediction by the object position prediction unit 10. Note that the operations of the object position prediction unit 10, the first object detection unit 30, and the second object detection unit 40 are not limited to these.

[0018] The object position prediction unit 10 uses position information indicating the past position of an object stored in the object position storage unit 50 as input for prediction. Based on the position information, the object position prediction unit 10 predicts the position of the object in the next image input from the image input unit 60. Hereinafter, the image input from the image input unit 60 will also be referred to as an input image. The object position prediction unit 10 may use position information of an object detected by the second object detection unit 40 as input for prediction. An example in which the object position prediction unit 10 uses position information of an object detected by the second object detection unit 40 as input for prediction will be described later.

[0019] The object detection target area determination unit 20 determines an area in the input image where the first object detection unit 30 should perform object detection processing, using the prediction results from the object position prediction unit 10, past position information of known objects stored in the object position storage unit 50, and information about new objects. Hereinafter, the area in the input image that is the target of object detection processing by the first object detection unit 30 is also referred to as the object detection target area. The object detection target area is, for example, a collection of partial areas that are part of the entire area of ​​the input image. Hereinafter, each individual partial area that makes up the object detection target area is also referred to as an individual object detection target area. Details of the individual object detection target area will be described later.

[0020] The first object detection unit 30 executes an object detection task using an image as input and outputs an object detection result. For example, the first object detection unit 30 uses only an area in an input image input from the image input unit 60 that includes the position of an object predicted by the object position prediction unit 10 (or an individual object detection target area determined by the object detection target area determination unit 20) as input. The first object detection unit 30 performs object detection processing using, for example, deep learning. The first object detection unit 30 holds a trained model. For example, a storage medium included in the first object detection unit 30 stores a trained model. The first object detection unit 30 performs inference by applying the trained model to the input. In addition, the first object detection unit 30 outputs a bounding box (BB), class, score, etc. for each object as an object detection result. However, the operation of the first object detection unit 30 is not limited to this. For example, the first object detection unit 30 may hold multiple trained models. The first object detection unit 30 may switch parameters used for inference (e.g., the trained model to be used and the input size) depending on the characteristics of the individual object detection target region (e.g., the height and width of the region). For example, the first object detection unit 30 may perform inference using a smaller input size for an individual object detection target region with a small height or width.

[0021] The second object detection unit 40 executes an object detection task on an input image input from the image input unit 60. The second object detection unit 40 outputs an object detection result obtained by executing the object detection task. The second object detection unit 40 performs object detection processing using deep learning, for example, similar to the first object detection unit 30.

[0022] The second object detection unit 40 performs object detection on a partial region of a new input image. For example, the second object detection unit 40 divides the input image into a plurality of regions and performs object detection by switching the target region for each input image.

[0023] The operation of the second object detection unit 40 will be described. FIG. 2 is an explanatory diagram illustrating an outline of the operation of the second object detection unit. Note that FIG. 2 is an explanatory diagram for facilitating understanding of the outline of the operation of the second object detection unit 40. Therefore, the operation of the second object detection unit 40 is not limited to that shown in FIG. 2.

[0024] FIG. 2 shows images of frames that are input consecutively as input images. FIG. 2(a) shows an image of frame i. FIG. 2(b) shows an image of frame i+1 that is input after frame i. FIG. 2(c) shows an image of frame i+2 that is input after frame i+1. FIG. 2(d) shows an image of frame i+3 that is input after frame i+2. FIG. 2(e) shows an image of frame i+4 that is input after frame i+3.

[0025] Figure 2 shows an example of dividing an input image into four regions and performing object detection by switching the target region for each input image. Specifically, this example shows an example in which the input image region is divided into two vertically and then two horizontally, resulting in four regions. That is, Figure 2 shows an example in which the input image is divided into four regions: the upper left region, the upper right region, the lower left region, and the lower right region.

[0026] In the example shown in FIG. 2, when an image of frame i is input as an input image, the second object detection unit 40 performs object detection on the upper left region of the input image as shown in FIG. 2(a). Next, when an image of frame i+1 is input as an input image, the second object detection unit 40 performs object detection on the upper right region of the input image as shown in FIG. 2(b). Next, when an image of frame i+2 is input as an input image, the second object detection unit 40 performs object detection on the lower left region of the input image as shown in FIG. 2(c). Next, when an image of frame i+3 is input as an input image, the second object detection unit 40 performs object detection on the lower right region of the input image as shown in FIG. 2(d). Next, when an image of frame i+4 is input as an input image, the second object detection unit 40 performs object detection on the upper left region of the input image as shown in FIG. 2(e).

[0027] In this way, the second object detection unit 40 divides the input image into a plurality of regions and performs object detection by switching the target region for each input image. Note that the method of dividing and switching the regions of the input image shown in Fig. 2 is one method that can be applied by the second object detection unit 40. The second object detection unit 40 can apply various other methods of dividing and switching the regions of the input image.

[0028] The second object detection unit 40 may divide the input image into a plurality of regions in a manner in which the regions partially overlap each other. For example, consider a case in which the input image is divided into a first region and a second region, and the first region and the second region are switched for each input image to be used as the target of object detection. In this case, the first region and the second region may partially overlap each other. Alternatively, the first region and the second region may not completely overlap each other.

[0029] The second object detection unit 40 can temporarily expand the area targeted for object detection. For example, when the second object detection unit 40 detects a new object at an edge of the area, the second object detection unit 40 temporarily expands the area adjacent to the edge.

[0030] Specifically, assume that the second object detection unit 40 performs object detection on the upper left region of the image of frame i shown in FIG. 2(a) and detects a new object at the bottom edge of the upper left region. Alternatively, assume that the second object detection unit 40 detects part of the new object at the bottom edge of the upper left region. That is, assume that the new object is predicted to be located at the boundary between the upper left region and the lower left region. In this case, when the second object detection unit 40 performs object detection on the lower left region of the image of frame i+2 shown in FIG. 2(c), it temporarily expands the lower left region to include the bottom edge of the upper left region. By doing so, the second object detection unit 40 can detect new objects appropriately depending on the situation.

[0031] The second object detection unit 40 may use a trained model different from that of the first object detection unit 30. For example, the second object detection unit 40 may use a trained model that is lighter in weight than the trained model used by the first object detection unit 30. By doing so, the image processing device 1 can reduce the processing load of the object detection process.

[0032] The second object detection unit 40 may intermittently perform object detection on a plurality of input images that are continuously input from the image input unit 60. For example, the second object detection unit 40 may perform object detection at predetermined intervals. Furthermore, for example, the second object detection unit 40 may perform object detection every predetermined number of frames.

[0033] For example, assume that the second object detection unit 40 is configured to perform object detection every four frames. In this case, when an image of frame i is input as an input image, the second object detection unit 40 performs object detection on the upper left region of the input image. Thereafter, the second object detection unit 40 does not perform object detection even when images of frames i+1 to i+3 are input sequentially. Then, when an image of frame i+4 is input as an input image, the second object detection unit 40 performs object detection on the upper right region of the input image. By doing so, the image processing device 1 can reduce the processing load of the object detection process.

[0034] The object position storage unit 50 stores information about objects (known objects) detected by the first object detection unit 30. The object position storage unit 50 also stores information about objects (new objects) detected by the second object detection unit 40. The object position storage unit 50 stores, as information about objects, all or part of the following: a bounding box, a class, a score, a detection time, an identifier of an input image in which the object was detected, and an identifier of the image input unit 60 that generated the input image.

[0035] The image input unit 60 generates an image to be processed by the image processing device 1 (i.e., an input image). The image input unit 60 may be realized by, for example, a surveillance camera. The image input unit 60 may use, as the input image, an image captured by the camera as is, or may use, as the input image, an image that has undergone preprocessing such as image processing or cropping. Furthermore, the image input unit 60 may input, as the input image, each image that is continuously output at a predetermined frame rate from a device external to the image processing device 1.

[0036] The object detection mode determination unit 70 determines the mode of execution of object detection by the second object detection unit 40. For example, the object detection mode determination unit 70 determines the mode of execution of object detection by the second object detection unit 40 to be the mode of the operation example shown in Fig. 2. The second object detection unit 40 performs object detection based on the result of the determination by the object detection mode determination unit 70.

[0037] The object detection mode determination unit 70 can also determine that object detection is not to be performed as the execution mode of object detection by the second object detection unit 40. In this case, the second object detection unit 40 does not perform object detection based on the determination result of the object detection mode determination unit 70. Such a configuration is applied, for example, to a configuration in which the second object detection unit 40 intermittently performs object detection on a plurality of input images that are input consecutively from the image input unit 60.

[0038] For example, object detection mode determination unit 70 may determine the execution mode of object detection by second object detection unit 40 for a new input image based on an object detection result related to a past input image. Details of the case where the execution mode of object detection is determined based on an object detection result related to a past input image will be described later.

[0039] Next, a method for predicting an object position by the object position predicting unit 10 will be described.

[0040] Methods for predicting the future position of an object from its past positions have been widely studied. For example, there is a research field called human trajectory prediction, which deals with predicting the position of a person. In recent years, methods for predicting human trajectory using deep learning have been widely studied. In deep learning methods, past movement trajectories are used as input, and inference is performed using a trained model to predict future movement trajectories.

[0041] The object position prediction unit 10 can perform object position prediction processing using any person trajectory prediction method. Many person trajectory prediction methods use information summarizing the movement trajectories of the same person as input to achieve high prediction accuracy. Furthermore, many person trajectory prediction methods perform tracking processing to identify the same person from multiple images captured at different times in the past in order to obtain information summarizing the movement trajectories of the same person. However, performing tracking processing involves a computational load. Therefore, applying a person trajectory prediction method that requires tracking processing to the image processing device 1 may increase the processing load and reduce processing throughput.

[0042] The object position prediction unit 10 can use a prediction method that does not require tracking processing instead of a prediction method that requires tracking processing. An example of a prediction method that does not require tracking processing will be described below. The object position prediction unit 10 of this embodiment uses a prediction method that does not require tracking processing, which will be described below. This prediction method is also referred to as a first prediction method of this embodiment. However, the prediction method that the object position prediction unit 10 can use is not limited to the first prediction method of this embodiment.

[0043] When using position information of an object at multiple different times as input to prediction, a tracking process may be performed to use information that compiles the movement trajectories of the same person. In this case, the input to prediction is not independent across the multiple times, and a dependency relationship may exist. On the other hand, there is a case where a tracking process is not performed, and only position information of an object at each time is used as input to prediction. In this case, the input to prediction is independent from one another across multiple times. The first prediction method of this embodiment is a prediction method that does not perform a tracking process. Therefore, the first prediction method of this embodiment uses information that is independent from one another across multiple different times as input to prediction.

[0044] The first prediction method of this embodiment uses position information of an object in an input image at a plurality of past times as input. In this example, the number of past (i.e., observed) times is Nobs. The first prediction method of this embodiment predicts the position of an object in the input image at the next time. However, the prediction target of the first prediction method of this embodiment is not limited to this. For example, the first prediction method of this embodiment may predict the position of an object in the input image at a plurality of future times. Furthermore, the number of objects included in the input image may be any number.

[0045] In the first prediction method of this embodiment, the input image and the output image (i.e., the prediction result) are divided into predetermined sizes, and the positions of the objects are managed in units of divided regions (grids). In the first prediction method of this embodiment, for example, if the original image is full HD (High Definition) [1920 × 1080] and the predetermined size is 32 × 32, management is performed using 60 × 34 grids.

[0046] In the first prediction method of this embodiment, position information of objects in Nobs past input images is used to create inference input data to be input to the trained model. The inference input data is, for example, a floating-point vector whose dimension is the number of grids. In this case, the inference input data contains a 1 in an element corresponding to a certain grid if one or more objects are present in that grid, and a 0 if no objects are present at all. The output from the trained model (i.e., the inference output data) is, for example, a floating-point vector whose dimension is the number of grids. In this case, the output from the trained model (i.e., the inference output data) contains a numerical value that is high when it is predicted that there is a high probability that an object exists in that grid, in an element corresponding to that grid.

[0047] In the first prediction method of this embodiment, for example, a predetermined threshold is used, and for each element of the inference output data, if the value is equal to or greater than the threshold, it is predicted that an object is present in the grid corresponding to that element. Furthermore, in the first prediction method of this embodiment, if the value is less than the threshold, it is predicted that an object is not present in the grid corresponding to that element. The trained model used in the first prediction method of this embodiment is generated by training a model for prediction (i.e., a prediction model) using the inference input data and inference output data generated from a ground truth dataset including images and position information related to moving objects.

[0048] The first prediction method of the present embodiment can use, for example, a deep learning model using a recurrent neural network (RNN) or a long short-term memory (LSTM) network. Furthermore, the first prediction method of the present embodiment can use a deep learning model using a convolutional neural network (CNN) or a transformer, using input obtained by combining past time-series data in the channel direction.

[0049] The first prediction method of this embodiment described above can predict the future positions of objects without identifying, separating, or tracking each individual object, even if multiple objects appear in an image at a given time. In other words, it can be said that each of the Nobs floating-point vectors is independent.

[0050] Next, a method for determining an object detection target region by object detection target region determining unit 20 will be described.

[0051] The object detection target area determination unit 20 extracts grids in which an object exists from the prediction results by the object position prediction unit 10 (i.e., information indicating whether an object is located in each grid). The object detection target area determination unit 20 also groups adjacent grids from the extracted grids to create a set of grids. However, the operation of the object detection target area determination unit 20 is not limited to this. The number of grid sets will be 0, 1, or multiple, depending on the predicted position of the object (group).

[0052] Object detection target region determination unit 20 may determine an area corresponding to a set of created grids as an individual object detection target region, and a set of the individual object detection target regions as an object detection target region. Alternatively, object detection target region determination unit 20 may determine, for each set of created grids, a rectangle of the smallest size that encompasses the area corresponding to that set, and determine the determined rectangle as the individual object detection target region.

[0053] The object detection target region determination unit 20 may adjust the individual object detection target region. For example, the object detection target region determination unit 20 may multiply the height or width of the individual object detection target region by a constant, or may add a constant value. By enlarging the individual object detection target region, it is possible to avoid oversights or defects when the prediction is incorrect.

[0054] For example, the object detection target area determination unit 20 checks whether or not the areas overlap for each pair of two individual object detection target areas. If there is an overlap, the object detection target area determination unit 20 may integrate the two individual object detection target areas (i.e., eliminate overlap) to form a single individual object detection target area. For example, the object detection target area determination unit 20 may determine the smallest rectangle that encompasses the two individual object detection target areas as the integrated individual object detection target area. By integrating the individual object detection target areas (eliminating overlap), it is possible to avoid double detection in object detection for each individual object detection target area. The object detection target area determination unit 20 may adjust the individual object detection target areas as described above after eliminating overlap.

[0055] When the object detection result by the second object detection unit 40 is available, the object detection target region determination unit 20 may update the existing object detection target region using the position information of the object detected by the second object detection unit 40. Furthermore, the object detection target region determination unit 20 may determine a new object detection target region using the position information of the object detected by the second object detection unit 40.

[0056] For example, for each detected object included in the object detection result by the second object detection unit 40, the object detection target area determination unit 20 checks whether the bounding box of the detected object is contained in the object detection target area. If the bounding box is not contained in the object detection target area, the object detection target area determination unit 20 may add the bounding box of the detected object as an individual object detection target area and update the object detection target area. In this case, the object detection target area determination unit 20 may adjust or integrate the individual object detection target areas as described above. The object detection target area determination unit 20 may treat an object that is included in the object detection result by the second object detection unit 40 and is not contained in the object detection target area as a new object. In this case, the object detection target area determination unit 20 may store information about the new object in the object position storage unit 50. The information about the new object may include, for example, position information such as a bounding box, as well as information that can identify the object as a new object.

[0057] When information about a new object is stored in object position storage unit 50, object detection target region determination unit 20 may update an existing object detection target region using the information about the new object. Alternatively, object detection target region determination unit 20 may determine a new object detection target region using the information about the new object.

[0058] For each new object, the object detection target region determination unit 20 may update the object detection target region by, for example, adding a region corresponding to the bounding box of the new object as an individual object detection target region. However, the new object may be moving. Therefore, the object detection target region determination unit 20 may adjust the bounding box of the new object before setting it as an individual object detection target region. For example, the object detection target region determination unit 20 may expand the bounding box by a predetermined value in all directions (i.e., add or subtract from the coordinate values). The object detection target region determination unit 20 may also adjust or integrate the individual object detection target regions described above.

[0059] The object detection target region determination unit 20 may update the information about the new object stored in the object position storage unit 50, using the object detection result for the input image input from the image input unit 60. The object detection result is expected to include the detection result of the new object at its latest position.

[0060] For example, for each new object, the object detection target region determination unit 20 searches for a detection result corresponding to the new object from among the object detection results. Then, the object detection target region determination unit 20 may update information about the new object stored in the object position storage unit 50 using the corresponding detection result. The object detection target region determination unit 20 may use any method as a method for searching for a detection result corresponding to the new object from among the object detection results. For example, the object detection target region determination unit 20 may calculate the IoU (Intersection over Union) between the new object and each object included in the object detection results, and determine that the object with the largest IoU corresponds to the new object. If the object detection target region determination unit 20 cannot find a detection result corresponding to the new object from among the object detection results (for example, if the largest IoU is less than a predetermined threshold), it may delete information about the new object. Alternatively, the object detection target region determination unit 20 may perform a tracking process for the new object and search for a detection result corresponding to the new object from among the object detection results.

[0061] The object detection target area determination unit 20 may delete information about a new object stored in the object position storage unit 50 at any timing. For example, the object detection target area determination unit 20 may delete information about a new object when the information about the new object has been accumulated sufficiently to be used as input for prediction by the object position prediction unit 10. The object detection target area determination unit 20 may determine that the information about the new object has been accumulated sufficiently to be used as input for prediction by the object position prediction unit 10 when the object detection process for the new object by the first object detection unit 30 has been executed Nobs times or more.

[0062] [Explanation of operation] Next, a description will be given of the operation of the image processing device 100 of this embodiment. Figures 3 and 4 are flowcharts illustrating the operation of the image processing device.

[0063] 3 and 4 show an example of the operation of object detection processing using a combination of first object detection unit 30 and second object detection unit 40. This operation includes an operation in which object detection target area determination unit 20 determines an object detection target area using the object detection result by second object detection unit 40 in addition to position information predicted based on position information of previously detected objects (known objects) stored in object position storage unit 50. This operation also includes an operation in which first object detection unit 30 performs object detection processing for each individual object detection target area that constitutes the object detection target area.

[0064] The image processing device 1 starts this operation, for example, every time an input image is input from the image input unit 60.

[0065] The object position prediction unit 10 acquires position information of objects (known objects) included in the most recent Nobs input images from the object position storage unit 50. Based on the acquired position information, the object position prediction unit 10 creates inference input data to be input to the predictive trained model of the first prediction method of this embodiment (step S100).

[0066] Next, the object position prediction unit 10 performs inference using the inference input data created in step S100 as input, and predicts the object position (step S101).

[0067] Next, object-detection target region determining unit 20 determines an object-detection target region based on the prediction result obtained in step S101 (that is, information indicating whether an object is located in each grid unit) (step S102).

[0068] Next, the object detection mode determination unit 70 determines the mode of object detection to be performed by the second object detection unit 40 (step S103). For example, the object detection mode determination unit 70 determines the mode of object detection to be performed by the second object detection unit 40 to be the mode of the operation example shown in FIG. 2. That is, the object detection mode determination unit 70 determines the mode of object detection to be performed in any one of the upper left region, upper right region, lower left region, and lower right region of the input image. Note that the object detection mode determination unit 70 can also determine that object detection is not to be performed as the mode of object detection to be performed by the second object detection unit 40. In this case, the processes of steps S104 to S105, which will be described later, are omitted.

[0069] Next, the second object detection unit 40 uses the input image input from the image input unit 60 as an input and performs object detection processing in the execution mode determined by the object detection mode determination unit 70 (step S104).

[0070] Next, object-detection target region determining unit 20 updates the object-detection target region determined in step S102 based on the object detection result obtained in step S104 (step S105). Step S105 will be described in detail later.

[0071] Next, for each individual object detection target area of ​​the object detection target area obtained in step S105 (step S106), first object detection unit 30 performs object detection processing on that area in the input image input from image input unit 60 (step S107). For example, first object detection unit 30 performs object detection processing using only the image corresponding to that area as input. The processing of step S107 is repeated until it has been performed on all individual object detection target areas. First object detection unit 30 converts bounding box information, which is position information of the object obtained as a result of the object detection processing, from coordinate values ​​in the individual object detection target area to coordinate values ​​in the input image input from image input unit 60.

[0072] Next, first object detection unit 30 stores the object position information obtained by the processes of steps S106 and S107 in object position storage unit 50 (step S108). First object detection unit 30 may also store information for identifying the input image, such as the time the input image was generated, an identifier of the input image, and an identifier of image input unit 60, in object position storage unit 50 together with the object position information. Furthermore, first object detection unit 30 may also store the input image itself in object position storage unit 50 at the same time.

[0073] Next, the object detection target area determination unit 20 stores information (e.g., bounding box information) about the new object that was processed in step S105C (details of which will be described later) included in step S105 in the object position storage unit 50 (step S109).

[0074] Next, step S105 shown in FIG. 3 will be described in detail with reference to FIG.

[0075] Object-detection target region determination section 20 performs the processes of steps S105B to S105C described below for each detected object obtained in step S104 (step S105A).

[0076] Object detection target region determination unit 20 checks whether the bounding box of the detected object is completely contained in the object detection target region determined in step S102 (step S105B). If it is completely contained (Yes in step S105B), object detection target region determination unit 20 ends the processing for the detected object.

[0077] If the detected object is not completely included (No in step S105B), object-detection target region determination unit 20 identifies the detected object as a new object. Furthermore, object-detection target region determination unit 20 updates the object-detection target region so that the bounding box of the detected object identified as a new object is completely included (step S105C).

[0078] If there are overlapping individual object detection target areas among the object detection target areas updated in steps S105A to S105C, object detection target region determining unit 20 merges them (step S105D).

[0079] 3 and 4 do not limit the operation of image processing device 1 of the present disclosure. For example, if a new object has been detected in the previous input image, the following process may be executed in step S109. That is, in step S109, object detection target region determination unit 20 searches for the new object stored in object position storage unit 50 and the object with the largest IoU from the object detection results. Thereafter, object detection target region determination unit 20 updates the information about the new object using the position information of the searched object.

[0080] For example, suppose the second object detection unit 40 detects an object as a new object from an input image, and then detects the object as a new object from Nobs input images including the input image. In this case, the position information for the object is sufficiently accumulated to be used as input for prediction by the first prediction method of this embodiment. Therefore, the object position prediction unit 10 and the object detection target area determination unit 20 of the image processing device 1 can perform processing on the object as a known object. In this case, the object detection target area determination unit 20 of the image processing device 1 may delete information about the object stored as a new object in the object position storage unit 50.

[0081] [Effect description] Next, the effects of this embodiment will be described. The image processing device 1 according to the first embodiment can reduce the processing load of the object detection process. Furthermore, the image processing device 1 according to the first embodiment can reduce the delay in the object detection process. The reasons for this are as follows.

[0082] An object position prediction unit 10 of an image processing device 1 according to the first embodiment predicts the position of an object included in an input image. An object detection target area determination unit 20 determines an individual object detection target area, which is a partial area of ​​the input image, based on the predicted object position. A first object detection unit 30 performs object detection processing for each individual object detection target area. The size of the individual object detection target area is expected to be smaller than that of the input image. That is, in the image processing device 1 according to the first embodiment, the amount of data to be subjected to the object detection processing is reduced, and the calculation load is accordingly reduced. Therefore, in the image processing device 1, the processing load of the object detection processing can be reduced. As a result, in the image processing device 1, the throughput of the object detection processing is expected to be improved. Furthermore, the delay in the object detection processing is expected to be reduced. That is, in the image processing device 1 according to the first embodiment, the period from when a target object appears in an image until it is detected is shortened (i.e., the delay is shortened), making it possible to quickly respond to the target object.

[0083] Furthermore, the second object detection unit 40 of the image processing device 1 performs object detection with the aim of preventing new objects from being overlooked. Then, the object detection target area determination unit 20 determines an individual object detection target area based on the prediction result by the object position prediction unit 10 and the object detection result by the second object detection unit 40 (e.g., this corresponds to the processing of step S105). With this configuration, the image processing device 1 can handle not only known objects detected in the past, but also new objects that have recently appeared. Furthermore, with this configuration, the image processing device 1 can also handle cases where a known object has been lost due to a prediction failure.

[0084] The image processing device 1 according to the first embodiment performs, on an input image, a prediction process by the object position prediction unit 10 and an object detection process by the first object detection unit 30, as well as an object detection process by the second object detection unit 40. Therefore, the image processing device 1 may have a higher processing load for the object detection process than a configuration in which the object detection process by the second object detection unit 40 is not performed.

[0085] Therefore, the second object detection unit 40 of the image processing device 1 according to the first embodiment performs object detection on a partial region of the new input image. With this configuration, the image processing device 1 can reduce the processing load compared to when the second object detection unit 40 performs object detection on the entire region of the new input image.

[0086] For example, the second object detection unit 40 may be configured to intermittently perform object detection on multiple consecutively input images, targeting the entire area of ​​the input image in order to prevent new objects from being overlooked. This configuration allows the image processing device 1 to reduce the processing load. However, in this case, the processing load increases only for the input images for which object detection by the second object detection unit 40 is performed, among the multiple input images, which may result in processing delays.

[0087] Therefore, the second object detection unit 40 of the image processing device 1 according to the first embodiment divides the input image into a plurality of regions and performs object detection by switching the target region for each input image. With this configuration, the image processing device 1 can suppress unevenness in the processing load for a plurality of input images and distribute processing delays.

[0088] [Variations] The object detection mode determination unit 70 may determine the execution mode of object detection by the second object detection unit 40 for a new input image, for example, based on object detection results for past input images. The object detection results for past input images include information such as prediction results by the object position prediction unit 10, object detection results by the first object detection unit 30, object detection results by the second object detection unit 40, and differences between predictions and object detection results. The object detection mode determination unit 70 estimates the tendency of new objects to appear based on the object detection results for past input images. Then, based on the estimation results, the object detection mode determination unit 70 determines the area, frequency, accuracy, etc. for object detection as the execution mode of object detection by the second object detection unit 40. The object detection mode determination unit 70 may determine the type of trained model to be used for object detection as the execution mode of object detection by the second object detection unit 40.

[0089] For example, if objects move from the left region to the right region in most cases in previously input images, the object detection mode determination unit 70 infers that new objects are likely to appear in the left region. Furthermore, if objects rarely appear in the upper region (e.g., the sky is captured) in previously input images, the object detection mode determination unit 70 infers that new objects are unlikely to appear in the upper region. Then, for example, the object detection mode determination unit 70 determines the execution mode of object detection by the second object detection unit 40 so that object detection is performed with high frequency in the region where new objects are likely to appear. Furthermore, for example, the object detection mode determination unit 70 determines the execution mode of object detection by the second object detection unit 40 so that object detection is performed with high accuracy in the region where new objects are likely to appear.

[0090] For example, if new objects are people in most cases in past input images, the object detection mode determination unit 70 infers that new objects of the person class are likely to appear. The object detection mode determination unit 70 then determines the execution mode of object detection by the second object detection unit 40, such that object detection is performed using a trained model specialized for detecting the person class. Alternatively, for example, if new objects are vehicles in most cases in past input images, the object detection mode determination unit 70 infers that new objects of the vehicle class are likely to appear. The object detection mode determination unit 70 then determines the execution mode of object detection by the second object detection unit 40, such that object detection is performed using a trained model specialized for detecting the vehicle class. Alternatively, for example, if new objects detected from past input images are of various object classes, the object detection mode determination unit 70 infers that there is no particular tendency for the object classes of new objects. The object detection mode determination unit 70 then determines the execution mode of object detection by the second object detection unit 40, such that object detection is performed using a trained model suitable for detecting various object classes. The object detection mode determination unit 70 may make different predictions according to the time period, day of the week, date and time, season, and the like.

[0091] With this configuration, the image processing device 1 can dynamically adjust parameters of object detection with the aim of preventing the second object detection unit 40 from overlooking a new object. As a result, the image processing device 1 can suitably perform object detection by the second object detection unit 40 according to the characteristics and circumstances of the input image.

[0092] The object detection target area determination unit 20 may determine an object detection target area including a prediction area where an object is predicted to be located and a prediction error corresponding area corresponding to the prediction error. For example, the object detection target area determination unit 20 determines the object detection target area by adding a prediction error corresponding area corresponding to the prediction error to a prediction area determined based on the prediction result by the object position prediction unit 10. The prediction error corresponding area is, for example, an area surrounding the prediction area. With this configuration, the image processing device 1 can reduce the possibility of missing object detection even if the prediction by the object position prediction unit 10 is incorrect.

[0093] The object detection target region determination unit 20 may set a prediction error corresponding region for a new input image based on, for example, object detection results for past input images. The object detection results for past input images include information such as prediction results by the object position prediction unit 10, object detection results by the first object detection unit 30, object detection results by the second object detection unit 40, and the difference between the prediction and the object detection result. The object detection target region determination unit 20 estimates the movement tendency of a known object based on the object detection results for the past input images. Then, the object detection target region determination unit 20 sets a prediction error corresponding region based on the estimation result.

[0094] For example, if an object moves from the left region to the right region in most cases of past input images, the object detection mode determination unit 70 sets a wide prediction error corresponding region to the right of the prediction region and a narrow prediction error corresponding region to the left of the prediction region. This configuration allows the image processing device 1 to dynamically adjust the parameters of the prediction error corresponding region corresponding to the prediction error. As a result, the image processing device 1 can optimally perform object detection by the first object detection unit 30 depending on the characteristics and situation of the input image.

[0095] The image processing device 1 according to the first embodiment can also be applied to images captured by multiple image capturing devices with different imaging directions. For example, consider a vehicle such as an automobile equipped with multiple cameras and LiDAR (Laser Imaging Detection and Ranging). Cameras are installed on the front, right, and left sides of the vehicle. In this case, an object captured by the front camera installed on the front side is captured by a side camera installed on the right or left side after a predetermined period of time. The image processing device 1 can perform image processing by utilizing such relationships between the cameras. Specifically, the image processing device 1 performs prediction using the object position prediction unit 10, object detection using the first object detection unit 30, and object detection for new objects using the second object detection unit 40 on the image captured by the front camera. The image processing device 1 then utilizes the results obtained from the image captured by the front camera to predict objects in the image captured by the side camera. In other words, even if an object is not yet captured in the image captured by the side camera, the image processing device 1 can calculate the position of the object in the side camera coordinate system from its positional relationship with the front camera and the object's position in the image captured by the front camera, and use this as input for prediction.

[0096] The image processing device 1 can also perform image processing using information acquired by the LiDAR. The information acquired by the LiDAR, although rough, can be used as an object detection result. Therefore, instead of performing object detection using the second object detection unit 40, the image processing device 1 may utilize the information acquired by the LiDAR to prevent new objects from being overlooked. The information acquired by the LiDAR also includes distance information. Therefore, the image processing device 1 may utilize the information acquired by the LiDAR for adjusting the prediction error correspondence region, etc. For example, the image processing device 1 may reduce the prediction error correspondence region for areas where the distance is far and increase the prediction error correspondence region for areas where the distance is close. The image processing device 1 may also utilize the information acquired by the LiDAR to adjust parameters of the object detection operation. For example, the image processing device 1 may use a lightweight object detection model or a high score threshold for areas where an object appears large. The image processing device 1 may use a high-precision object detection model or a low score threshold for areas where an object appears small or where multiple objects are close or overlapping. The image processing device 1 can utilize information acquired by various types of sensors other than LiDAR. That is, the image processing device 1 can perform the image processing described using a camera or LiDAR as an example, by utilizing information acquired by various types of sensors.

[0097] The image processing device 1 can also perform image processing using vehicle movement information. The vehicle movement information includes, for example, the vehicle's movement speed and steering angle. The image processing device 1 may adjust various parameters of the processing performed by each unit of the image processing device 1 depending on whether the vehicle is moving or stopped. The image processing device 1 may also adjust various parameters of the processing performed by each unit of the image processing device 1 depending on the vehicle's movement direction. For example, when the vehicle is turning right, the object detection mode determination unit 70 of the image processing device 1 sets a wide prediction error corresponding area on the right side of the prediction area and a narrow prediction error corresponding area on the left side of the prediction area. The image processing device 1 may also switch the trained model used for prediction, etc. depending on whether the vehicle is traveling straight or turning. The image processing device 1 may also correct the prediction result or object detection result based on the vehicle movement information.

[0098] In the above description, an example has been shown in which the time interval applied to the input and output for prediction by the object position prediction unit 10 is the same as the interval at which the input image is generated by the image input unit 60. However, the present disclosure is not limited to this. Each unit constituting the image processing device 1, including the object position prediction unit 10 and the image input unit 60, may be configured to operate at different time periods. For example, each unit constituting the image processing device 1 may operate using the latest information available at the time each unit operates.

[0099] The first object detection unit 30 and the second object detection unit 40 may output a score (or confidence or accuracy) for each detected object as an object detection result. When using the object detection results, each unit constituting the image processing device 1 may filter the detection results using a predetermined threshold and score. The predetermined threshold may differ depending on the application.

[0100] In the above description, an example has been given in which the trained model used in the first prediction method of this embodiment is used as a correct answer dataset containing images and position information related to moving objects, and the inference input data and inference output data are generated from the correct answer dataset to train a model for prediction (prediction model). However, the present disclosure is not limited to this. When training the model, the correct answer position information contained in the correct answer dataset may be transformed, processed, or augmented. For example, similar to adjusting the object detection target area, the object position may be enlarged up, down, left, right, or left. While such training may result in a prediction result that is larger than the actual object position, it is expected to reduce the effect of overlooking object detection. Furthermore, within the Nobs frame used for a single prediction, each object may be translated, flipped left and right or up, rotated, or scaled. Furthermore, the object may be scaled in the time direction. That is, the moving speed of the object may be slowed or accelerated. For example, Nobs frames may be extracted by thinning out one frame from 2·Nobs frames of the correct answer data and used as input for prediction. In this case, the object in that frame will move twice as fast.

[0101] In the above description, an example has been described in which the object position prediction unit 10 applies the first prediction method of the present embodiment to all known objects. However, the present disclosure is not limited to this. The object position prediction unit 10 may apply a prediction method different from the first prediction method of the present embodiment to some objects. The object position prediction unit 10 may apply, for example, a prediction method involving a tracking process. Generally, a prediction method involving a tracking process increases the processing load, but on the other hand, improves prediction accuracy. Therefore, using a prediction method involving a tracking process is expected to improve detection accuracy (e.g., reducing oversight due to prediction failure). The object position prediction unit 10 may apply a different position prediction method to objects whose score, identified as an object detection result, is lower than a predetermined threshold. Furthermore, the object position prediction unit 10 may divide the input image into multiple regions and apply a different prediction method to each region, or may switch prediction parameters for each region. Region configuration information and switching patterns may be provided in advance.

[0102] In the above description, an example has been shown in which both the second object detection unit 40 and the first object detection unit 30 perform object detection processing in the operation of the object detection processing shown in FIG. 3 . However, the present disclosure is not limited to this. For example, in the operation shown in FIG. 3 , the processing of step S105 and the object detection processing by the first object detection unit 30 (steps S106 and S107) may be omitted. In that case, the object detection result by the second object detection unit 40 (step S104) may be treated as the object detection result by the first object detection unit 30. Furthermore, in step S104, instead of the object detection processing by the second object detection unit 40, the object detection processing by the first object detection unit 30 may be performed on the entire input image.

[0103] In the above description, an example has been described in which an input image used as an input for prediction and an input image to be subjected to object detection (or an input image to be output as a prediction result) are input from the same image input unit. However, the present disclosure is not limited to this. For example, the image processing device 1 may be configured to include multiple image input units 60 (e.g., image input unit 60A and image input unit 60B). In this case, the object position prediction unit 10 may use the position of an object in the input image input from the image input unit 60A as an input for prediction to predict the future position of the object in the input image input from the image input unit 60B. The trained model used for inference by the object position prediction unit 10 may be trained assuming such a configuration. The difference in the shooting range (or angle of view) between the image input unit 60A and the image input unit 60B may be fixed, for example. Furthermore, information indicating the difference in the shooting range (or angle of view) between the image input unit 60A and the image input unit 60B may be used during training.

[0104] The above description illustrates an example in which the image processing device 1 executes an object detection task. However, the present disclosure is not limited to this. The image processing device 1 may execute other tasks, such as pose estimation or region recognition (segmentation). For example, the first object detection unit 30 may execute other tasks in addition to or instead of the object detection task. Furthermore, the second object detection unit 40 may execute other tasks in addition to or instead of the object detection task. If the task does not directly generate object position information, the image processing device 1 may generate object position information or alternative information required for prediction operations by the object position prediction unit 10 based on the output from the task. For example, if the task is a pose estimation task, information (type, position, etc.) regarding the articulation points of a person in the input image may be obtained as the output. The image processing device 1 may generate a person rectangle from the obtained articulation point group and use the rectangle as object position information.

[0105] Furthermore, for example, the first object detection unit 30 may execute an image classification task instead of an object detection task. Generally, an image classification task has a smaller processing load than an object detection task and is expected to improve throughput. Furthermore, the first object detection unit 30 may switch whether or not to use the image classification task depending on the characteristics of the individual object detection target area. For example, the first object detection unit 30 may select the image classification task in the following cases: When the size of the individual object detection target area is less than a predetermined threshold When the number of objects present in the individual object detection target area is expected to be 1 or less - When integration (reduction of overlap) processing is not applied to the individual object detection target area

[0106] In the above description, an example has been given in which the image processing device 1 executes the object detection processing operation shown in FIG. 3 every time an input image is input from the image input unit 60. However, the present disclosure is not limited to this. For example, if the image processing device 1 can determine in advance that no known object exists and that the object detection target area obtained as a result of the prediction is empty, the image processing device 1 may omit execution of this operation. This is expected to reduce the load associated with the prediction.

[0107] In the above description, an example has been given in which the image input unit 60 generates an input image. However, the present disclosure is not limited to this. The image input unit 60 may receive compressed image data from an external device and decode the image data to generate an input image. The image input unit 60 may perform decoding of a compression format such as JPEG (Joint Photographic Experts Group) or MPEG (Moving Picture Experts Group). Furthermore, the image input unit 60 may switch generation methods using prediction results for past input images.

[0108] For example, if it can be determined in advance that no known object exists and that the object detection target area obtained as a result of the prediction is empty, the image input unit 60 may not perform decoding processing, or may perform decoding processing using a low-load, low-quality decoding method. Furthermore, the image input unit 60 may perform decoding processing only on a portion of the object detection target area. For example, the image input unit 60 may perform decoding processing on each individual object detection target area, or may perform decoding processing on only the smallest rectangular area that includes all individual object detection target areas. The image input unit 60 may fill areas that have not been decoded with a dummy image (e.g., a blacked-out image). Furthermore, the image input unit 60 may output area information for areas that have not been decoded to components that use the input image, and have the components use the input image by referring to the area information. Furthermore, when it is time to perform object detection processing using the second object detection unit 40 shown in FIG. 3 , the image input unit 60 may generate an input image as usual, or may generate an input image using a low-load, low-quality decoding method.

[0109] [Hardware configuration] In the above description, an example has been used in which the object position prediction unit 10, the object detection target area determination unit 20, the first object detection unit 30, the second object detection unit 40, the object position storage unit 50, the image input unit 60, and the object detection mode determination unit 70 are included in the same device (image processing device 1). However, the first embodiment is not limited to this.

[0110] For example, the image processing device 1 may be configured by connecting devices having functions corresponding to the respective components via a predetermined network.

[0111] Each component of the image processing device 1 may be configured as a hardware circuit, or multiple components of the image processing device 1 may be configured as a single piece of hardware.

[0112] Alternatively, the image processing device 1 may be realized as a computer device including a CPU, a read-only memory (ROM), and a random access memory (RAM). The image processing device 1 may be realized as a computer device that further includes an input / output connection circuit (IOC) in addition to the above configuration. The image processing device 1 may be realized as a computer device that further includes a network interface circuit (NIC) in addition to the above configuration.

[0113] Alternatively, the image processing device 1 may be realized as a computer device further including an arithmetic unit that performs calculations for part or all of the processes related to tracking, such as calculation of feature amounts and inference.

[0114] FIG. 5 is a block diagram showing the configuration of an information processing device 600, which is an example of the hardware configuration of the image processing device 1.

[0115] The information processing device 600 includes a CPU 610, an arithmetic unit 611, a ROM 620, a RAM 630, an internal storage device 640, an IOC 650, and a NIC 680. The information processing device 600 constitutes a computer device.

[0116] The CPU 610 loads a program from the ROM 620 and / or the internal storage device 640. Based on the loaded program, the CPU 610 controls the RAM 630, the internal storage device 640, the arithmetic unit 611, the IOC 650, and the NIC 680. The computer device including the CPU 610 controls these components and realizes the functions of the object position prediction unit 10, the object detection target area determination unit 20, the first object detection unit 30, the second object detection unit 40, and the object position storage unit 50.

[0117] When implementing each function, the CPU 610 may use the RAM 630 or the internal storage device 640 as a temporary storage medium for the program.

[0118] Furthermore, the CPU 610 may read a program contained in a storage medium 690 that stores the program in a computer-readable manner, using a storage medium reading device (not shown). Alternatively, the CPU 610 may receive a program from an external device (not shown) via the NIC 680, store the program in the RAM 630 or the internal storage device 640, and operate based on the stored program.

[0119] The arithmetic unit 611 may be, for example, any of a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and an AI (Artificial Intelligence) chip. The arithmetic unit 611 may perform calculations for part or all of processes such as object detection and prediction inference under the control of a program executed by the CPU 610. Data, programs, circuit information, and the like required for the arithmetic unit 611 to execute calculations may be stored in, for example, the ROM 620, the RAM 630, the internal storage device 640, and the like.

[0120] The ROM 620 stores fixed data and programs executed by the CPU 610. The ROM 620 is, for example, a P-ROM (Programmable ROM) or a flash ROM.

[0121] The RAM 630 temporarily stores programs and data executed by the CPU 610. The RAM 630 is, for example, a D-RAM (Dynamic-RAM).

[0122] The internal storage device 640 stores data and programs that are to be saved long-term by the information processing device 600. The internal storage device 640 may operate as the object position storage unit 50. The internal storage device 640 may also operate as a temporary storage device for the CPU 610. The internal storage device 640 is, for example, a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), or a disk array device.

[0123] The ROM 620 and the internal storage device 640 are non-volatile (non-transitory) recording media. On the other hand, the RAM 630 is a volatile (transitory) recording media. The CPU 610 can operate based on a program stored in the ROM 620, the internal storage device 640, or the RAM 630. In other words, the CPU 610 can operate using a non-volatile recording medium or a volatile recording medium.

[0124] The IOC 650 mediates data between the CPU 610 and the input device 660 and display device 670. The IOC 650 is, for example, an IO interface card or a USB (Universal Serial Bus) card. Furthermore, the IOC 650 is not limited to being connected via a wired connection like a USB, and may also be connectable wirelessly.

[0125] The input device 660 is a device that receives instructions from an operator of the information processing device 600. For example, the input device 660 receives parameters. The input device 660 is, for example, a keyboard, a mouse, or a touch panel. The input device 660 may also be an input device that functions as the image input unit 60. The image input unit 60 may be, for example, a camera device.

[0126] The display device 670 is a device that can display information to the operator of the information processing device 600. The display device 670 is, for example, a liquid crystal display, an organic electroluminescence display, or electronic paper.

[0127] The NIC 680 relays data exchange with external devices (not shown) via a network. The NIC 680 is, for example, a LAN (Local Area Network) card. The NIC 680 may be connected to external devices wirelessly, rather than by wire.

[0128] The information processing device 600 configured in this manner can achieve the same effects as the image processing device 1. This is because the CPU 610 of the information processing device 600 can realize the same functions as the image processing device 1 based on a program. Also, the CPU 610 and the arithmetic unit 611 of the information processing device 600 can realize the same functions as the image processing device 1 based on a program.

[0129] <Second embodiment> Next, a second embodiment of the present disclosure will be described. An image processing device 1B according to the second embodiment generates an aggregated image by collecting image regions of object detection target regions, and performs object detection on the aggregated image.

[0130] The second embodiment will be described with reference to the drawings. In the drawings referred to in describing the second embodiment, the same reference numerals are used to designate components that perform the same operations as those in the first embodiment. Detailed descriptions of these components will be omitted.

[0131] [Configuration Description] The configuration of an image processing device 1B according to the second embodiment will be described with reference to the drawings. Note that, similar to the first embodiment, the image processing device 1B may be configured using a computer device such as that shown in FIG.

[0132] FIG. 6 is a block diagram showing an example of the configuration of an image processing device 1B according to the present disclosure.

[0133] The image processing device 1B illustrated in FIG. 6 includes an object position prediction unit 10, an object detection target area determination unit 20, a first object detection unit 30B, a second object detection unit 40, an object position memory unit 50, an image input unit 60, an object detection mode determination unit 70, and an aggregate image generation unit 80.

[0134] The aggregated image generation unit 80 generates an aggregated image by collecting image regions of the object detection target region. The aggregated image generation unit 80 receives information indicating the object detection target region from the object detection target region determination unit 20, and generates an image (aggregated image) by collecting image regions of individual object detection target regions. The process of collecting (i.e., copying) the obtained image regions is called packing. The aggregated image generation unit 80 may generate one or more aggregated images. When generating the aggregated image, the aggregated image generation unit 80 associates the individual object detection target regions with information indicating the arrangement of the individual object detection target regions on the aggregated image and stores them in the storage unit. When packing, the aggregated image generation unit 80 may provide a gap (spacing) of a predetermined width between each image region.

[0135] When performing packing, the aggregated image generation unit 80 may change the size of the individual object detection target region (i.e., enlarge or reduce it) and copy it to the aggregated image. Reducing the size may reduce the number of aggregated images, potentially shortening the inference processing time for object detection. Increasing the size may improve recognition accuracy. The aggregated image generation unit 80 may determine whether to change the size of the individual object detection target region and determine the size after the change based on a predetermined threshold value, etc. For example, the aggregated image generation unit 80 may determine whether to change the size of the individual object detection target region and determine the size after the change based on the area of ​​the individual object detection target region. When changing the size of the individual object detection target region, the aggregated image generation unit 80 may perform image processing such as interpolation processing. Furthermore, the aggregated image generation unit 80 may perform any image processing in addition to or instead of changing the size of the image region. The aggregated image generation unit 80 may perform image processing such as brightness adjustment, luminance adjustment, color adjustment, contrast adjustment, geometric correction, etc., as image processing.

[0136] First object detection unit 30B has the same function as first object detection unit 30 of the first embodiment. However, first object detection unit 30B uses an aggregate image generated by aggregate image generation unit 80 as an input, instead of using an image of an individual object detection target area as an input.

[0137] [Explanation of operation] Next, an example of the operation of the image processing device 1B according to the second embodiment will be described with reference to the drawings. Among the operations (steps) of the image processing device 1B according to the second embodiment, the same operations (steps) as those of the image processing device 1 according to the first embodiment will be assigned the same step numbers. Further, detailed descriptions of these operations (steps) will be omitted.

[0138] FIG. 7 is a flowchart showing an example of the operation of the object detection process in the image processing device 1B according to the present disclosure.

[0139] The image processing device 1B performs the processes of steps S100 to S105.

[0140] Next, the aggregate image generating unit 80 generates an aggregate image based on the object detection target region obtained in step S105 (step S200).

[0141] Next, for each aggregated image generated in step S200 (step S201), first object detection unit 30B performs object detection processing on the aggregated image (step S202). For example, first object detection unit 30B executes the object detection processing using the aggregated image as input. First object detection unit 30B converts bounding box information, which is position information of the object obtained as a result of the object detection processing, from coordinate values ​​in the aggregated image to coordinate values ​​in the input image input from image input unit 60. The processing of step S202 is repeated until it has been executed on all generated aggregated images.

[0142] [Effect description] Next, the effects of the second embodiment will be described.

[0143] The image processing device 1B according to the second embodiment can reduce the processing load of the object detection process, similar to the first embodiment. Furthermore, the image processing device 1B according to the second embodiment can reduce the delay in the object detection process.

[0144] The image processing device 1B generates an aggregated image based on the individual object detection target region. Next, the image processing device 1B performs object detection for each generated aggregated image. The size of the aggregated image is expected to be smaller than the input image. Therefore, in the image processing device 1B according to the second embodiment, the amount of target data for the object detection process is reduced, and the calculation load is accordingly reduced. Therefore, in the image processing device 1B, the processing load of the object detection process can be reduced. As a result, in the image processing device 1B, it is expected that the throughput of the object detection process will be improved. In addition, it is expected that the delay in the object detection process will be shortened. That is, in the image processing device 1B according to the second embodiment, the period from when the target object appears in the image until it is detected will be shortened (i.e., the delay will be shortened), making it possible to quickly respond to the target object.

[0145] Next, an overview of the present disclosure will be described. Fig. 8 is a block diagram showing an overview of an image processing device according to the present disclosure. Image processing device 100 (in the embodiment, image processing device 1 or image processing device 1B) shown in Fig. 8 includes object position prediction means 110 (in the embodiment, implemented by object position prediction unit 10) that predicts the position of an object in a new input image based on the position of the object detected in a past input image, second object detection means 120 (in the embodiment, implemented by second object detection unit 40) that performs object detection on a partial area in the new input image, object detection target area determination means 130 (in the embodiment, implemented by object detection target area determination unit 20) that determines an object detection target area to be the target of object detection in the new input image based on the prediction result by the object position prediction means 110 and the object detection result by the second object detection means 120, and first object detection means 140 (in the embodiment, implemented by first object detection unit 30 or first object detection unit 30B) that performs object detection on the object detection target area determined by the object detection target area determination means 130. With this configuration, the image processing device 100 can reduce the processing load of the object detection process.

[0146] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Each embodiment can be combined with other embodiments as appropriate.

[0147] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but also to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0148] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0149] (Appendix 1) an object position prediction means for predicting a position of an object in a new input image based on the position of the object detected in a past input image; a second object detection means for detecting an object in a partial area of ​​a new input image; an object detection target area determination means for determining an object detection target area to be a target for object detection in a new input image based on a prediction result by the object position prediction means and an object detection result by the second object detection means; a first object detection means for detecting an object in the object detection target area determined by the object detection target area determination means; 1. An image processing device comprising:

[0150] (Appendix 2) The second object detection means divides an input image into a plurality of regions and performs object detection by switching the target region for each input image. 2. The image processing device of claim 1.

[0151] (Appendix 3) The second object detection means performs object detection using a trained model that is lighter than the trained model used by the first object detection means. 3. The image processing device according to claim 1 or 2.

[0152] (Appendix 4) The second object detection means intermittently performs object detection on a plurality of input images that are input consecutively. 4. An image processing device according to any one of claims 1 to 3.

[0153] (Appendix 5) an object detection mode determination means for determining an execution mode of object detection by the second object detection means; The second object detection means performs object detection based on the determination result of the object detection mode determination means. 5. An image processing device according to any one of claims 1 to 4.

[0154] (Appendix 6) The object detection mode determination means determines an execution mode of object detection by the second object detection means for a new input image based on an object detection result for a past input image. 6. The image processing device according to claim 5.

[0155] (Appendix 7) The object detection target area determining means determines the object detection target area including a prediction area in which an object is predicted to be located and a prediction error corresponding area corresponding to a prediction error. 7. An image processing device according to any one of Supplementary Note 1 to Supplementary Note 6.

[0156] (Appendix 8) The object detection target region determining means sets the prediction error corresponding region for a new input image based on an object detection result for a past input image. 8. The image processing device of claim 7.

[0157] (Appendix 9) The computer predicting a position of an object in a new input image based on a position of the object detected in a previous input image; Perform object detection on a part of the area in the new input image, determining an object detection target region in the new input image to be subjected to object detection based on the prediction result and the object detection result; Object detection is performed in the determined object detection target area. An image processing method comprising:

[0158] (Appendix 10) On the computer, an object position prediction process for predicting the position of an object in a new input image based on the position of the object detected in a past input image; a second object detection process for detecting an object in a partial region of the new input image; an object detection target area determination process that determines an object detection target area to be a target for object detection in a new input image based on the prediction result of the object position prediction process and the object detection result of the second object detection process; a first object detection process for performing object detection on the object detection target area determined in the object detection target area determination process; Image processing program to be executed.

[0159] (Appendix 11) A computer-readable recording medium storing an image processing program, The image processing program is installed on a computer. an object position prediction process for predicting the position of an object in a new input image based on the position of the object detected in a past input image; a second object detection process for detecting an object in a partial region of the new input image; an object detection target area determination process that determines an object detection target area to be a target for object detection in a new input image based on the prediction result of the object position prediction process and the object detection result of the second object detection process; a first object detection process for detecting an object in the object detection target area determined in the object detection target area determination process; Recording medium.

[0160] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 8 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 9, 10, and 11 in the same dependency relationship as Supplementary Notes 2 to 8. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods. [Explanation of symbols]

[0161] 1, 1B, 100 Image processing device 10 Object position prediction unit 20 Object detection target area determination unit 30, 30B First object detection unit 40 Second object detection unit 50 Object position memory section 60 Image input unit 70 Object detection mode determination unit 80 Aggregate image generation unit 110 Object position prediction means 120 second object detection means 130 Object detection target area determination means 140 first object detection means 600 Information Processing Devices 610 CPU 611 arithmetic unit 620 ROM 630 RAM 640 Internal storage 650 IOC 660 Input Devices 670 Display equipment 680 NIC 690 Storage medium

Claims

1. an object position prediction means for predicting a position of an object in a new input image based on the position of the object detected in a past input image; a second object detection means for detecting an object in a partial area of ​​the new input image; an object detection target area determination means for determining an object detection target area to be a target of object detection in a new input image based on a prediction result by the object position prediction means and an object detection result by the second object detection means; a first object detection means for detecting an object in the object detection target area determined by the object detection target area determination means; 1. An image processing device comprising:

2. The second object detection means divides an input image into a plurality of regions, and performs object detection by switching the target region for each input image.

2. The image processing device according to claim 1.

3. The second object detection means performs object detection using a trained model that is lighter than the trained model used by the first object detection means.

3. The image processing device according to claim 1.

4. The second object detection means intermittently detects objects from a plurality of input images that are input consecutively.

3. The image processing device according to claim 1.

5. an object detection mode determination means for determining an execution mode of object detection by the second object detection means; The second object detection means performs object detection based on the determination result of the object detection mode determination means.

3. The image processing device according to claim 1.

6. The object detection mode determining means determines an execution mode of object detection by the second object detection means for a new input image based on an object detection result for a past input image.

6. The image processing device according to claim 5.

7. The object detection target area determining means determines the object detection target area including a prediction area in which an object is predicted to be located and a prediction error corresponding area corresponding to a prediction error.

3. The image processing device according to claim 1.

8. The object detection target region determining means sets the prediction error corresponding region for a new input image based on an object detection result for a past input image.

8. The image processing device according to claim 7.

9. The computer predicting a position of an object in a new input image based on a position of the object detected in a previous input image; Perform object detection on a part of the area in the new input image, determining an object detection target region in the new input image to be subjected to object detection based on the prediction result and the object detection result; Object detection is performed in the determined object detection target area. An image processing method comprising:

10. On the computer, an object position prediction process for predicting the position of an object in a new input image based on the position of the object detected in a past input image; a second object detection process for detecting an object in a partial region of the new input image; an object detection target area determination process that determines an object detection target area to be a target for object detection in a new input image based on a prediction result of the object position prediction process and an object detection result of the second object detection process; a first object detection process for performing object detection on the object detection target area determined in the object detection target area determination process; Image processing program to be executed.

Images