Information processing apparatus, information processing method, and program

Abstract

To easily create an image in which objects are difficult to be detected, for a device that detects objects from an image.SOLUTION: An information processing apparatus has: detection means that detects objects from a picked-up image; tracking means that tracks the objects on time-series picked-up images on the basis of a result of the detection of the objects; estimation means that, on the basis of the result of detection of the objects and a result of tracking of the objects, estimates, from the picked-up image, an area in which the detection means has difficulty in detecting the objects from the picked-up image, for each type of the objects; and creation means that creates an image obtained by superimposing a predetermined object image corresponding to the types of objects on a predetermined background image on the basis of a result of the estimation.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing technique for generating an image. [Background technology]

[0002] Conventionally, there is known a system for monitoring traffic by capturing images of objects such as vehicles and people moving on the road using a network camera, analyzing the captured images, and detecting the type and position of the objects in the images.To realize a highly accurate traffic monitoring system, it is desirable that the detection device that detects the type and position of objects in the images through image analysis is a device that is optimized for the environment where the images are captured.

[0003] As a method for detecting objects from captured images, Non-Patent Document 1 discloses a method that uses deep learning technology to train a network model in advance on the type and coordinates of objects in an image, enabling the type and coordinates of objects in unknown images to be detected. To realize a detection device optimized for the local environment using such a method, it is desirable to collect images of difficult scenes where object detection failures or false detections are likely to occur, and train the network model. However, creating training data for the network model requires collecting multiple images of these difficult scenes and then assigning type and coordinate information to each object shown in each image, which requires a lot of manual work.

[0004] In response to this, Patent Document 1 discloses a technology that uses a simulation system that uses CG images to facilitate the generation of training data for various images that are difficult for a detection device to detect. By using the technology described in Patent Document 1, it is possible to easily create training data based on images of scenes that are difficult for a detection device to detect.

[0005] Furthermore, Patent Document 2 discloses a technology for detecting weak areas where false detection or non-detection may occur in a detection device using a neural network model, and estimating information about the weak areas. By using the technology described in Patent Document 2 to estimate areas that may be weak scenes for the detection device, it is possible to use the technology to create effective training data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-76992 [Patent Document 2] Japanese Patent Publication No. 2022-169068 [Non-patent literature]

[0007] [Non-Patent Document 1] J. Redmon, A. Farhadi, “YOLO9000:Better Faster Stronger”, Computer Vision and Pattern Recognition (CVPR) 2016. Summary of the Invention [Problem to be solved by the invention]

[0008] However, even if the technologies described in Patent Documents 1 and 2 are used, it is difficult to generate images to be used as training data in order to realize a detection device that can detect various objects that may exist in various positions in the real environment without error or omission.

[0009] Therefore, an object of the present invention is to make it possible to easily generate an image in which it is difficult for a device that detects an object from an image to detect an object. [Means for solving the problem]

[0010] The information processing device of the present invention is characterized by having a detection means for detecting an object from a captured image, a tracking means for tracking the object on a time series of captured images based on the detection result of the object by the detection means, an estimation means for estimating an area from the captured image in which it is difficult for the detection means to detect the object for each type of object based on the detection result of the object by the detection means and the tracking result of the object by the tracking means, and a generation means for generating an image in which a predetermined object image corresponding to the type of object is superimposed on a predetermined background image based on the estimation result by the estimation means. [Effects of the Invention]

[0011] According to the present invention, it is possible to easily generate an image in which it is difficult for a device that detects an object from the image to detect an object. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example of a system configuration. [Figure 2] FIG. 1 is a diagram illustrating an example of a schematic internal configuration of an imaging device. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging apparatus. [Figure 4] FIG. 2 illustrates an example of a hardware configuration of a server. [Figure 5] FIG. 2 illustrates an example of a functional configuration of a server. [Figure 6] FIG. 10 is a diagram illustrating an example of a precision region image. [Figure 7] FIG. 10 is a diagram showing an example of a precision region image including a region where no object exists. [Figure 8] FIG. 10 is a diagram showing an example of an object existence region setting screen. [Figure 9] FIG. 10 is a diagram illustrating an example of object existence region information. [Figure 10] 10 is a flowchart showing the flow of information processing according to the present embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of object detection result information. [Figure 12]FIG. 10 is a diagram illustrating an example of object tracking result information. [Figure 13] 10 is a flowchart of precision region estimation processing according to the first embodiment. [Figure 14] FIG. 10 is a diagram used to explain precision region estimation processing. [Figure 15] FIG. 10 is a diagram illustrating an example of precision region information according to the first embodiment. [Figure 16] 10 is a flowchart of an image generation process according to the first embodiment. [Figure 17] 10A and 10B are diagrams illustrating an example of input data and output data of an image generation process. [Figure 18] 10A and 10B are diagrams illustrating an example of an accuracy region confirmation screen and an accuracy region setting screen. [Figure 19] 10 is a flowchart of precision region estimation processing according to the second embodiment. [Figure 20] 13 is a flowchart of precision region estimation processing according to the third embodiment. [Figure 21] FIG. 13 is a diagram illustrating an example of precision region information according to the third embodiment. [Figure 22] 13 is a flowchart showing the flow of information processing according to the fourth embodiment. [Figure 23] FIG. 13 is a diagram illustrating an example of a prompt used in precision region estimation according to the fourth embodiment. [Figure 24] 13 is a flowchart of precision region estimation processing according to the fifth embodiment. [Figure 25] 10 is a flowchart of a detailed parameter estimation process. [Figure 26] FIG. 10 is a diagram illustrating an example of detailed parameter information. [Figure 27] 13 is a flowchart of an image generation process according to the fifth embodiment. [Figure 28] FIG. 13 is a diagram illustrating an example of a prompt used in image generation according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configuration of each embodiment may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, in the following embodiments, the same or similar configurations and processing steps are designated by the same reference numerals, and redundant explanations will be omitted.

[0014] [First embodiment] <System configuration> FIG. 1 is a diagram showing an example of the configuration of an image analysis system 101 according to this embodiment. The following description will be given taking an example in which this image analysis system 101 is applied to a traffic monitoring system. However, the image analysis system 101 of this embodiment is not limited to traffic management systems, and can be applied to any system that analyzes images and outputs predetermined information. The image analysis system 101 shown in FIG. 1 is configured to include an imaging device 110, a network 120, and a server 130.

[0015] The imaging device 110 is an imaging device such as a network camera. In this embodiment, the imaging device 110 is exemplified as having a built-in arithmetic device capable of processing images, but is not limited to this. For example, an external information processing device such as a PC (personal computer) connected to the imaging device may process images, and a combination of the imaging device and the external information processing device may be treated as the imaging device 110. The server 130 is an information processing device such as a PC, and has information processing functions including the image analysis processing according to this embodiment. Furthermore, the server 130 is a device that can receive input from a user and output information to the user (for example, display information, etc.).

[0016] The imaging device 110 and the server 130 are connected to each other via a network 120 so that they can communicate with each other. The network 120 includes a plurality of routers, switches, cables, and the like that comply with a communication standard such as Ethernet (registered trademark). In this embodiment, the network 120 may be any network that enables communication between the imaging device 110 and the server 130, and may be constructed with any scale, configuration, and conformance to any communication standard. For example, the network 120 may be the Internet, a wired local area network (LAN), a wireless LAN, a wide area network (WAN), or the like. Furthermore, for example, the network 120 may be configured to enable communication using a communication protocol that complies with the ONVIF (Open Network Video Interface Forum) standard. These are merely examples, and the network 120 may also be configured to enable communication using other communication protocols, such as a proprietary communication protocol.

[0017] <Configuration of imaging device> Next, a description will be given of the configuration of the imaging device 110. Fig. 2 is a diagram showing a schematic configuration of the imaging device 110. The imaging device 110 includes, for example, an imaging unit 201, an image processing unit 202, an arithmetic processing unit 203, and a distribution unit 204. It is assumed that each of the components shown in Fig. 2 is configured by hardware such as a circuit.

[0018] The imaging unit 201 includes an imaging element that captures a formed optical image and outputs an analog signal, a lens system for forming an optical image of a subject or the like on the imaging element, and an optical system driver. The lens system includes a zoom lens that changes the angle of view, a focus lens that adjusts the focus, and an aperture that adjusts the amount of light, and the optical system driver drives the zoom lens, focus lens, aperture, and the like. The imaging element also has a gain function that adjusts the sensitivity when converting light into an analog signal. These functions are adjusted based on setting values ​​notified by the image processing unit 202. The analog signal acquired by the imaging unit 201 is converted into a digital signal by an analog-to-digital conversion circuit (not shown) and transferred to the image processing unit 202 as an image signal.

[0019] The image processing unit 202 is configured to include an image processing engine and its peripheral devices. The peripheral devices include, for example, a RAM (Random Access Memory) and drivers for each I / F (Interface). The image processing unit 202 generates image data by performing predetermined image processing such as development processing, filtering, sensor correction, and noise removal on the image signal acquired from the imaging unit 201. The image processing unit 202 also transmits setting values ​​to the optical system driving unit and the imaging element, and performs angle of view adjustment and exposure adjustment so that an appropriately exposed image with a desired angle of view can be acquired. The image data generated in the image processing unit 202 is transferred to the arithmetic processing unit 203.

[0020] The arithmetic processing unit 203 is composed of one or more processors such as a CPU or MPU, memories such as RAM or ROM, drivers for each I / F, etc. Note that CPU is an acronym for Central Processing Unit, MPU for Micro Processing Unit, RAM for Random Access Memory, and ROM for Read Only Memory.

[0021] The distribution unit 204 includes a network distribution engine and peripheral devices such as a RAM and an ETH PHY module. The ETH PHY module is a module that executes processing at the physical (PHY) layer of Ethernet. The distribution unit 204 converts the image data and processing result data acquired from the arithmetic processing unit 203 into a format that can be distributed to the network 120, and outputs the converted data to the network 120.

[0022] 3 is a diagram showing an example of the functional configuration of the imaging device 110. The imaging device 110 includes an imaging control unit 301, a signal processing unit 302, a storage unit 303, a control unit 304, an analysis unit 305, and a communication unit 306. The imaging control unit 301 includes the imaging unit 201 described above, controls the imaging operation of the imaging unit 201 , and transmits the imaging signal obtained by the imaging unit 201 to the signal processing unit 302 .

[0023] The signal processing unit 302 includes the image processing unit 202 and the arithmetic processing unit 203, and generates captured image data by performing predetermined processing on the imaging signal sent from the imaging control unit 301, and also encodes the captured image data. Hereinafter, this captured image data will be referred to as a captured image or simply an image. If the captured image is a still image, the signal processing unit 302 encodes the still image using an encoding method such as JPEG (Joint Photographic Experts Group). If the captured image is a moving image, the signal processing unit 302 encodes the moving image using an encoding method such as H.264 / MPEG-4 AVC or HEVC (High Efficiency Video Coding). The signal processing unit 302 can also encode the moving image using an encoding method selected by a user from a plurality of preset encoding methods via an operation unit (not shown) of the imaging device 110.

[0024] The storage unit 303 stores temporary data during various processes. The control unit 304 controls the signal processing unit 302, the storage unit 303, the analysis unit 305, and the communication unit 306 so that they each perform the corresponding predetermined processing. The analysis unit 305 performs various image analysis processes on the captured image. The communication unit 306 includes the distribution unit 204 described above, and communicates with the server 130 via the network 120 .

[0025] <Server configuration> FIG. 4 is a diagram showing an example of the hardware configuration of the server 130. The server 130 is configured by an information processing device such as a general PC. That is, as shown in FIG. 4, the server 130 includes a processor 401 such as a CPU, memories such as a RAM 402 and a ROM 403, a large-capacity storage device 404 such as an HDD or SSD, and a communication I / F 405. The server 130 can perform various functions by having the processor 401 execute various programs, including the information processing program according to this embodiment, stored in the ROM 403 or the large-capacity storage device 404. The RAM 402 is used as a temporary storage area when the processor 401 performs various processes. The communication I / F 405 is connected to the network 120 and communicates with external devices such as the imaging device 110.

[0026] 5 is a diagram showing an example of the functional configuration of the server 130 according to this embodiment. The server 130 includes, as its functional configuration, for example, a communication unit 501, a control unit 502, a display unit 503, an operation unit 504, a setting unit 505, a storage unit 506, a detection unit 507, a tracking unit 508, an accuracy estimation unit 509, and an image generation unit 510.

[0027] The communication unit 501 includes the above-mentioned communication I / F 405, and communicates with an external device such as the image capturing device 110, for example, via the network 120. Note that this is merely an example, and the communication unit 501 can also establish a direct connection with the image capturing device 110 and communicate with the image capturing device 110, for example, without going through the network 120 or another device. The display unit 503 presents various information to the user via, for example, a screen display on a built-in or external display device. In this embodiment, the display unit 503 also has a browser function, and presents various information to the user by displaying the results rendered by the browser on the screen of the display device. Details of the information presented to the user via the screen display of the display device will be described later. The operation unit 504 accepts operations from the user. In this embodiment, the operation unit 504 is a mouse, a keyboard, or the like, which the user operates to input user operations via the browser described above. Of course, this is not limiting, and the operation unit 504 may be any other device capable of receiving instructions from the user, such as a touch panel or a microphone.

[0028] The control unit 502 controls the communication unit 501, display unit 503, operation unit 504, setting unit 505, storage unit 506, detection unit 507, tracking unit 508, accuracy estimation unit 509, and image generation unit 510 to each perform the processing described below.

[0029] The setting unit 505 executes setting processing, which will be described later. The detection unit 507 functions as an object detector that detects detection target objects (hereinafter referred to as detection target objects) that appear in captured images sent from the image capture device 110. As an example, the detection unit 507 performs object detection processing using an object detector having an object detection model trained by machine learning that applies deep learning technology. As will be described in detail later, when the detection unit 507 detects detection target objects from captured images, it acquires at least an object class that indicates the type of object for each detected object and the vertex coordinates of a circumscribing rectangle (a rectangle called a bounding box) for each object. The detection unit 507 then sends the object detection results from the object detection processing to the control unit 502. The control unit 502 stores the object detection results from the detection unit 507 in the storage unit 506 as object detection result information.

[0030] The tracking unit 508 executes an object tracking process to track an object on the captured image of each successive frame in time series. As will be described in detail later, the tracking unit 508 performs the tracking process based on a circumscribing rectangle (bounding box) detected by object detection processing from the captured image of the current frame and a circumscribing rectangle (bounding box) detected previously and tracked by object tracking processing. The tracking unit 508 then sends the object tracking result of the object tracking processing to the control unit 502. The control unit 502 stores the object tracking result by the tracking unit 508 in the storage unit 506 as object tracking result information. In this embodiment, the circumscribing rectangle of an object detected by the object detection processing of the detection unit 507 will be called the "detected rectangle," and the circumscribing rectangle of an object tracked by the object tracking processing of the tracking unit 508 will be called the "tracked rectangle."

[0031] The accuracy estimation unit 509 executes a process for estimating, for each type (object class) of detection target object, an area where object detection is difficult when the object detector of the detection unit 507 detects an object from a captured image, that is, an area where false detection or detection omission is likely to occur.The accuracy estimation unit 509 then determines the estimation result of the process for estimating an area where object detection is difficult for each type (object class) of detection target object as the estimated accuracy.That is, for each object class of the detection target object, the accuracy estimation unit 509 determines a partial area where object detection is estimated to be difficult from a captured image as an area with high estimation accuracy, and on the other hand, determines a partial area where object detection is estimated to be easy as an area with low estimation accuracy.

[0032] Furthermore, the accuracy estimation unit 509 performs rendering processing to generate an image representing the estimation accuracy for each partial region estimated for each type (object class) of the detection target object. In the following description, an image representing the estimation accuracy for each partial region estimated for each type of detection target object is referred to as an accuracy region image. In other words, the accuracy region image represents a higher estimation accuracy for regions where object detection is estimated to be more difficult, and an image representing a lower estimation accuracy for regions where object detection is estimated to be less difficult.

[0033] The image generation unit 510 executes an image generation process using the accuracy region image generated by the accuracy estimation unit 509, a pre-registered background image, and a prompt (described later) as input data, to generate a generated image and correct answer data (described later). The image generation unit 510 of this embodiment generates a generated image by superimposing a predetermined object image corresponding to the estimation accuracy of a partial region for each type of object to be detected in the accuracy region image on a background image. Furthermore, when generating a generated image, the image generation unit 510 preferentially arranges the predetermined object image in the region with the highest estimation accuracy obtained by the accuracy estimation unit 509.

[0034] In this embodiment, the image generation unit 510 performs image generation processing using an image generator of an image generation model trained by machine learning, which finds hidden patterns in large amounts of data through repeated calculations. Therefore, in this embodiment, the predetermined image corresponding to the estimation accuracy of the partial region for each type of detection target object is an image of an object to be detected by an object detector in an area where object detection is difficult, and is an image of the object used for machine learning. Note that, as the image generation model, for example, a model applying the technology described in the following reference document can be used.

[0035] References: Alan D. Thompson,'Inside language models (from GPT-4 to PaLM)

[0036] The storage unit 506 stores the object detection model of the object detector, object presence area information, object detection result information, object tracking result information, precision area images for each object class, the image generation model of the image generator, background images, generated images, correct answer data, etc. Details of the information held by the storage unit 506 will be described later. The background image is an image captured in advance by the image capture device 110, and is an image captured in an environment where no detection target object is present within the angle of view of the image capture device 110. As an example, when the image capture device 110 captures a specific road, the background image is an image that does not capture detection target objects such as vehicles (and may include trucks, vans, motorcycles, etc.) or people, and only captures the road and surrounding buildings. However, without being limited to this, it is also possible to remove detection target objects from the image captured by the image capture device 110 using an image editing tool or the like, generate an image consisting only of the road and surrounding buildings, etc., and use this as the background image.

[0037] <Accuracy area image> In this embodiment, the accuracy region image is an image rendered based on the estimation accuracy calculated by an accuracy region estimation process that estimates an area in a captured image where object detection is difficult, and is an image in which the estimation accuracy is expressed, for example, by different colors. FIG. 6 is a diagram showing an example of an accuracy region image 600 rendered based on the estimation accuracy resulting from estimation of an area of ​​a certain object class. In the example of FIG. 6, the color differences are expressed by shades of gray, with areas expressed with darker shades of gray representing areas where object detection is difficult (i.e., high estimation accuracy), and areas expressed with white representing areas where object detection is not difficult (i.e., low estimation accuracy). That is, in the example of FIG. 6, areas 601, 602, and 603 each represent areas where object detection is difficult. Furthermore, in this embodiment, the darker the shade of gray (the higher the estimation accuracy), the more difficult the area where object detection is. In the example of FIG. 6, area 602 is darker gray than area 601, and area 603 is darker gray than area 602. Therefore, it can be seen that the estimation accuracy is higher in the region 602 than in the region 601, and furthermore, the region 603 is higher than the region 602, that is, the region is more difficult to detect an object.

[0038] Fig. 7 is a diagram showing an example of a precision area image 700 different from the example of Fig. 6. The precision area image 700 is a diagram in which an area not set as an object existence area in the object existence area setting process described later, that is, an area 701 where no object exists, is superimposed on the precision area image 600 shown in Fig. 6. Note that the precision area image 700 may be used on a precision area confirmation screen transitioning from the precision area setting screen of Fig. 18 described later.

[0039] <Object existence area setting screen> Fig. 8 is a diagram showing an example of an object presence area setting screen generated and displayed by setting unit 505 according to this embodiment. A user of server 130 can set an object presence area through the object presence area setting screen as shown in Fig. 8. Object presence area setting screen 800 includes an area where a captured image is displayed and where the user can set an object presence area, an area where the user inputs an object class name of an object to be detected, and an area where buttons to be operated by the user are displayed.

[0040] In the example of Fig. 8, object class name input form 807 is an area for the user to input the object class name of the detection target object. However, the values ​​that can be input in object class name input form 807 must correspond to the object classes of the detection target objects that can be detected by detection unit 507. Fig. 8 shows an example in which "vehicle" is input as the object class name in object class name input form 807.

[0041] The captured image display area 801 is an area where an image captured by the imaging device 110 is displayed, and is an area where the user can set an object presence area. The captured image displayed in the captured image display area 801 is an image captured by the imaging device 110 of an intersection where a person 802, a vehicle 803, a vehicle 804, and a vehicle 805 are present. In this embodiment, the object presence area is set by the user manually inputting an object presence area while viewing the captured image displayed in the captured image display area 801. In this embodiment, the user can manually input the object presence area as an area formed by a polygon for the captured image in the captured image display area 801. The setting unit 505 acquires the coordinates of each vertex of the object presence area formed by the polygon manually input by the user, and acquires coordinate information of the object presence area represented by the vertex coordinates as area coordinate information. Note that the user may manually input multiple polygonal object presence areas, and the object presence areas of each object class may overlap. In the example of Figure 8, for the object class indicated by "vehicle" entered in the object class name input form 807, an area represented by grid lines in the captured image display area 801 is set as the object existence area 806.

[0042] The setting button 808 and the OK button 809 are buttons that the user can press by clicking the mouse or touching. When the user presses the setting button 808 by clicking the mouse, touching, or the like, the setting unit 505 sets the object existence area set by the user in the captured image display area 801 as the object existence area corresponding to the object class set by the user in the object class name input form 807. Then, the setting unit 505 sends information about the object class and coordinate information (area coordinate information) of the object existence area corresponding to that object class to the control unit 502. The control unit 502 stores the object class information and area coordinate information of the detection target object in the storage unit 506 as object existence area information. Also, when the user presses the OK button 809 by clicking or touching the mouse, the control unit 502 causes the accuracy estimation unit 509 to perform accuracy region estimation processing, as described below, and further causes the image generation unit 510 to perform image generation processing.

[0043] <Object existence area information> 9 is a diagram showing an example of object existence region information 900 set on the object existence region setting screen and stored in the storage unit 506. The object existence region information 900 includes coordinates (X, Y) 901 and object class information 902. The coordinates (X, Y) 901 are a set of X and Y coordinate values, and each X and Y coordinate value corresponds to each pixel coordinate of the captured image in the captured image display area 801. In other words, if the captured image is an image of 1080 × 720 pixels, the coordinates (X, Y) 901 store 1080 × 720 records.

[0044] The object class information 902 stores an object class name associated with each X and Y coordinate of the coordinates (X, Y) 901, and stores information corresponding to the object class name entered in the object class name input form 807 of the object existence area setting screen 800. That is, according to the object existence area information 900, an object class name in the object class information 902 is associated with each X and Y coordinate of the coordinates (X, Y) 901, thereby representing an object existence area for each object class. Note that, among the X and Y coordinates of the coordinates (X, Y) 901, an area consisting of X and Y coordinates for which an object class name is not stored in the object class information 902 is not set as an object existence area.

[0045] <Processing flow> The flow of information processing executed by the server 130 of this embodiment will be described below. Each processing process (processing step) shown in each flowchart below is realized by the processor 401 of the server 130 executing an information processing program deployed on the RAM 402. However, this is merely an example, and some or all of each processing process in the flowcharts described below may be executed not only by the server 130 but also by the imaging device 110 or dedicated hardware. Note that in each flowchart below, the symbol S represents a processing process (processing step).

[0046] <Information Processing According to the First Embodiment> Fig. 10 is a flowchart showing the overall flow of information processing executed in each functional unit of the server 130 shown in Fig. 5. The processing of this flowchart is triggered by, for example, the user pressing the OK button 809 on the object existence region setting screen 800.

[0047] First, in S1001, the control unit 502 controls the communication unit 501 to acquire a captured image from the imaging device 110. Note that the captured image acquired in S1001 may be an image that has been captured by the imaging device 110 and stored in the mass storage device 404 in advance.

[0048] Next, in S1002, the detection unit 507 executes object detection processing on the captured image acquired in S1001 to detect detection target objects appearing in the image. When the detection unit 507 detects detection target objects from the captured image, it acquires an object class indicating the type of the detected object (hereinafter referred to as the detected object) and the vertex coordinates of a detection rectangle (vertex coordinates of a bounding box) for each detected object, and sends these to the control unit 502. The control unit 502 stores information on the object class and vertex coordinates of the detection rectangle acquired for each detected object in the storage unit 506 as object detection result information.

[0049] 11 is a diagram showing an example of object detection result information that is the result of object detection processing by detection unit 507. Object detection result information 1100 includes information corresponding to each detected object: detection ID 1101, object class 1102, detection reliability 1103, and bounding box coordinates 1104. An identifier for identifying each detected object is stored in the detection ID 1101. The identifier stored in the detection ID 1101 is composed of a random character string of alphabets, numbers, etc. The object class 1102 stores the object class name of the detected object.

[0050] The detection reliability value indicating the reliability of the object detection result for each detected object is stored in the detection reliability 1103. The value indicating the detection reliability is, for example, a numerical value expressed in the range from 0 to 1, and the closer the numerical value is to 1, the higher the detection reliability is. The vertex coordinates of the bounding box, which is the detection rectangle for each detected object, are stored in the bounding box coordinates 1104. The coordinate values ​​stored in the bounding box coordinates 1104 are coordinate values ​​in a coordinate system with the upper left coordinate of the captured image as the origin. The bounding box coordinates 1104 store the X and Y coordinates of the upper left corner and the X and Y coordinates of the lower right corner of the detection rectangle (bounding box) of the detected object.

[0051] Next, in S1003, tracking unit 508 acquires the object tracking result by performing object tracking processing using the detected object result information detected from the image of the current frame and the object detection result information that is previous in time series among the object detection result information stored in storage unit 506. Then, tracking unit 508 sends the object tracking result obtained by the object tracking processing to control unit 502, and control unit 502 stores the object tracking result in storage unit 506 as object tracking result information.

[0052] Here, in the object tracking process, the object detection result information obtained in the object detection process for the image of the current frame is associated with the object tracking result information for each object obtained in the previous object tracking process in time series. More specifically, the tracking unit 508 predicts the position of each object in the image of the current frame based on the tracking result information of the object (hereinafter referred to as the tracked object) tracked in the object tracking process performed in the previous frame in time series. Then, if a detected object detected in the image of the current frame corresponds to the predicted position obtained in the prediction process, the tracking unit 508 assigns the same identifier to the detected object corresponding to the predicted position as the tracked object in the previous frame in time series. In other words, the tracking unit 508 assigns the same identifier to the detected object corresponding to the predicted position among the detected objects detected in the image of the current frame and the tracked object previous in time series when the predicted position was calculated, as they are the same object. On the other hand, the tracking unit 508 assigns a new identifier to the detected objects detected in the image of the current frame that do not correspond to the predicted position. In addition, if there is a predicted position among the predicted positions predicted by the tracking unit 508 that does not correspond to the detected object detected from the image of the current frame, the control unit 502 stores an identifier corresponding to that predicted position in the storage unit 506.

[0053] 12 is a diagram showing an example of object tracking result information obtained by object tracking processing by the tracking unit 508. Object tracking result information 1200 includes, for each tracked object, a tracking ID 1201, an object class 1202, a tracking reliability 1203, bounding box coordinates 1204, a corresponding detection ID 1205, and a number of lost detections 1206. The tracking ID 1201 stores an identifier assigned to each tracked object in the past. If the detected object detected in the current frame and the previous tracked object in the time series are the same object, the tracking unit 508 assigns the same tracking ID to the detected object as the tracking ID assigned to the previous tracked object in the time series. The tracking ID is composed of a random string of letters, numbers, etc.

[0054] The object class 1202 stores the object class name that indicates the type of each tracked object. Information indicating the reliability of object tracking for each object by the tracking unit 508 is stored in the tracking reliability 1203. A numerical value expressed in the range of 0 to 1, for example, is used as the information indicating the tracking reliability.

[0055] The bounding box coordinates 1204 store the vertex coordinates of a bounding box, which is a tracking rectangle for each past tracked object. Similar to the bounding box coordinates 1104 in Fig. 11, the bounding box coordinates 1204 store the coordinates (Z, Y) of the upper left corner and the coordinates (X, Y) of the lower right corner of the tracking rectangle (bounding box), with the upper left corner of the captured image as the origin. Note that the bounding box coordinates 1204 store the coordinates of the tracking rectangle that have been fine-tuned based on the positional relationship between the coordinates of the detection rectangle of the detected object detected by the detection unit 507 and the coordinates of the predicted position predicted by the tracking unit 508.

[0056] The detection ID of a detected object associated with a previously tracked object as the same object, that is, the detection ID of a detected object to which the same tracking ID as the tracked object has been assigned, is stored in the corresponding detection ID 1205. On the other hand, if there is no detected object associated with a previously tracked object as the same object, an identifier indicating that there is no detected object associated with the previously tracked object as the same object, for example, "None", is stored in the corresponding detection ID 1205.

[0057] The number of lost detections 1206 stores a numerical value representing the number of images (number of frames) in which the tracking object could not be associated with the detected object. In other words, the number of lost detections 1206 stores a numerical value representing the number of images (number of frames) in which a detected object identical to the object being tracked did not exist. For example, if a detected object identical to the tracking object does not exist in the image of the current frame, the number of lost detections 1206 stores a value obtained by adding "1" to the already stored numerical value. Also, if a detected object identical to the tracking object does exist in the image of the current frame, the number of lost detections 1206 stores a numerical value of "0".

[0058] Returning to the explanation of the flowchart in FIG. After the above-mentioned S1003, in the next S1004, the accuracy estimation unit 509 executes an accuracy region estimation process to estimate a region in the captured image where object detection is difficult, based on the object detection result information, object tracking result information, and object presence region information stored in the storage unit 506. Then, the accuracy estimation unit 509 generates an accuracy region image based on the accuracy region estimation result obtained by the accuracy region estimation process, and sends the accuracy region image to the control unit 502. The control unit 502 stores the accuracy region image in the storage unit 506. Note that the accuracy estimation unit 509 generates an accuracy region image as the accuracy region estimation result, but may also generate region information in which, for example, coordinates and corresponding accuracy information values ​​are stored, in addition to the accuracy region image.

[0059] Next, in S1005, the control unit 502 determines whether an instruction to stop the application (stop execution of the program) or to stop image capture by the imaging device 110 has been input by the user or the like. If an instruction to stop the application or stop the imaging device 110 has not been input, the control unit 502 returns the process to S1001 and controls to repeat the processes of S1001 to S1004 described above. On the other hand, if an instruction to stop the application or stop the imaging device 110 has been input, the control unit 502 stops the series of processes from S1001 to S1004 and proceeds to S1006. In S1006, the image generating unit 510 executes the image generating process described below.

[0060] <Processing flow of precision region estimation according to the first embodiment> FIG. 13 is a detailed flowchart of the accuracy range estimation process executed by the accuracy estimation unit 509 in S1004 of FIG. The accuracy estimation unit 509 sequentially performs loop processing of S1301 to S1309 for each record corresponding to each tracked object in the object tracking result information 1200 stored in the storage unit 506. The processing of the flowchart in Fig. 13 is performed using the object detection result information 1100 and object tracking result information 1200 stored in the storage unit 506. For this reason, in the following description, each record corresponding to each tracked object in the object tracking result information 1200 will be referred to as an object tracking result, and each record corresponding to each detected object in the object detection result information 1100 will be referred to as an object detection result. The processing of S1302 to S1308 in the flowchart in Fig. 13 will be described below with reference to Fig. 14.

[0061] Images 1410, 1420, 1430, and 1440 in Fig. 14(a) are examples of images captured in time series over a certain period of time, acquired from the image capture device 110. These images 1410 to 1440 show vehicles 803, 804, and 805, respectively, which are moving slowly, and Fig. 14(a) shows the results of object detection processing and object tracking processing performed on each of the vehicles 803 to 805.

[0062] In image 1410, detection rectangle 1411 represents the object detection result for vehicle 803, and tracking rectangle 1412 represents the object tracking result for vehicle 803. Similarly, detection rectangle 1413 represents the object detection result for vehicle 804, tracking rectangle 1414 represents the object tracking result for vehicle 804, detection rectangle 1415 represents the circumscribing rectangle of the object detection result for vehicle 805, and tracking rectangle 1416 represents the object tracking result for vehicle 805. Similarly, detection rectangles and tracking rectangles for vehicles in the images are also shown for images 1420 to 1440.

[0063] Here, in the case of image 1420, detection rectangle 1422 represents the object detection result for vehicle 804, and tracking rectangle 1423 represents the object tracking result for vehicle 804. On the other hand, in the case of image 1420, tracking rectangle 1421 for vehicle 803 is shown, but the detection rectangle is not. In other words, in the case of image 1420, it is indicated that object detection for vehicle 803 has not been possible. Similarly, tracking rectangle 1424 for vehicle 805 is shown, but the detection rectangle is not shown, so it is indicated that object detection for vehicle 805 has not been possible.

[0064] Furthermore, in the case of image 1430, detection rectangle 1432 represents the object detection result for vehicle 804, and tracking rectangle 1433 represents the object tracking result for vehicle 804. On the other hand, in the case of image 1430, tracking rectangle 1431 for vehicle 803 is shown, but no detection rectangle is shown, indicating that object detection has not been performed for vehicle 803. Similarly, tracking rectangle 1434 for vehicle 805 is shown, but no detection rectangle is shown, indicating that object detection has not been performed for vehicle 805.

[0065] In addition, in the case of image 1440, detection rectangle 1441 represents the object detection result for vehicle 803, and tracking rectangle 1442 represents the object tracking result for vehicle 803. Similarly, detection rectangle 1443 represents the object detection result for vehicle 804, and tracking rectangle 1444 represents the object tracking result for vehicle 804. On the other hand, in the case of image 1440, tracking rectangle 1445 for vehicle 805 is shown, but no detection rectangle is shown, indicating that object detection for vehicle 805 has not been performed.

[0066] Precision region images 1450, 1460, 1470, and 1480 in FIG. 14(b) are examples of precision region images generated by the precision estimation unit 509 based on the object detection results and object tracking results for each of the images 1410 to 1440 shown in FIG. 14(a).

[0067] The precision region image 1450 is an image rendered based on the estimated precision calculated by precision region estimation processing using the object detection results and object tracking results of the image 1410. In the case of the image 1410, as described above, object detection and object tracking were successful for each of the vehicles 803 to 805, and object detection within the captured image was not difficult, so a precision region with low estimation precision has been estimated.

[0068] The precision region image 1460 is an image created by updating the precision region image 1450 based on the object detection results and object tracking results of the image 1420. The precision region image 1460 shows an example of a region in the captured image where object detection is difficult, that is, a precision region 1461 and a precision region 1462 with high estimation accuracy, which have been estimated. The precision region 1461 is estimated based on the object detection results and object tracking results of the vehicle 803, and the precision region 1462 is estimated based on the object detection results and object tracking results of the vehicle 805.

[0069] Precision region image 1470 is an image created by updating precision region image 1460 based on the object detection results and object tracking results of image 1430. Precision region image 1470 shows an example in which precision region 1463 and precision region 1464, each with high estimated accuracy, have been estimated as regions where object detection is difficult. Precision region 1463 has been estimated based on the object detection results and object tracking results of vehicle 803, and precision region 1464 has been estimated based on the object detection results and object tracking results of vehicle 805. Precision regions 1463 and 1464 have been created taking into consideration precision regions 1461 and 1462 estimated in precision region image 1460. Precision region 1461 is shown to be a region where object detection is difficult, with a higher estimated accuracy than precision region 1463, and precision region 1462 is shown to be a region where object detection is difficult, with a higher estimated accuracy than precision region 1464.

[0070] Precision region image 1480 is an image created by updating precision region image 1470 based on the object detection results and object tracking results of image 1440, and shows an example in which precision region 1465 and precision region 1466, each with high estimated accuracy, are estimated. Precision region 1465 is estimated based on the object detection results and object tracking results of vehicle 803, and precision region 1466 is estimated based on the object detection results and object tracking results of vehicle 805. Precision regions 1465 and 1466 are created taking into consideration precision regions 1463 and 1464 estimated in precision region image 1470. Precision region 1464 is shown to be a region where object detection is more difficult, with a higher estimated accuracy than precision region 1466. In the case of precision region 1465, precision region 1463 estimated in precision region image 1470 has been updated, and precision regions 1461, 1463, and 1465 are shown as regions with similar estimated accuracy.

[0071] Returning to the explanation of the flowchart in FIG. In S1302, the accuracy estimation unit 509 determines whether the tracking rectangle of the object tracking result is within the object existence area. In this embodiment, the accuracy estimation unit 509 determines whether the tracking rectangle of the object tracking result is within the object existence area corresponding to the relevant object class 1202. More specifically, the accuracy estimation unit 509 calculates the midpoint coordinate of the bottom point of the tracking rectangle of the object tracking result based on the vertex coordinates of the bounding box coordinates 1204 corresponding to the object tracking result. Then, the accuracy estimation unit 509 determines that the tracking rectangle of the object tracking result is within the object existence area if the relevant object class 1202 and the object class information 902 match at the coordinates (X, Y) 901 of the object existence area information 900 corresponding to the midpoint coordinate. In addition to this method, the accuracy estimation unit 509 may perform a similar determination by, for example, calculating the center coordinate of the tracking rectangle of the object tracking result based on the vertex coordinates of the bounding box coordinates 1204 corresponding to the object tracking result.

[0072] Alternatively, for example, the accuracy estimation unit 509 may calculate the area of ​​the tracking rectangle of the object tracking result from the bounding box coordinates 1204, and determine whether the tracking rectangle of the object tracking result is within the object presence area based on the area ratio of the area of ​​the tracking rectangle that overlaps with the object presence area. For example, if the area ratio of the area of ​​the tracking rectangle that overlaps with the object presence area is equal to or greater than a predetermined ratio threshold, the accuracy estimation unit 509 determines that the tracking rectangle of the object tracking result is within the object presence area. If the accuracy estimation unit 509 determines, as a result of the determination process in S1302 described above, that the tracking rectangle of the object tracking result is within the object existence area, it proceeds to the process in S1303; on the other hand, if it determines that it is outside the object existence area, it proceeds to the process for the next object tracking result.

[0073] In S1303, the accuracy estimation unit 509 determines whether a corresponding detection ID 1205 exists for the object tracking result. For example, the accuracy estimation unit 509 determines whether the information of the corresponding detection ID 1205 is information (for example, "None") indicating that no object detection result exists corresponding to the object tracking result. If the accuracy estimation unit 509 determines that a corresponding detection ID 1205 does not exist, that is, if the corresponding detection ID 1205 is "None", the accuracy estimation unit 509 proceeds to processing of S1306, whereas if it determines that a corresponding detection ID 1205 exists, the accuracy estimation unit 509 proceeds to processing of S1304.

[0074] In S1304, the accuracy estimation unit 509 determines whether the number of lost detections 1206 in the object tracking result information 1200 corresponding to the previous frame is 1 or greater. For example, the accuracy estimation unit 509 determines whether the number of lost detections 1206 corresponding to the object tracking result of the previous frame, which has the same tracking ID as the tracking ID 1201 of the object tracking result of the current frame, is a numerical value of 1 or greater. If the accuracy estimation unit 509 determines that the number of lost detections 1206 is 1 or greater, the process proceeds to S1307. On the other hand, if the accuracy estimation unit 509 determines that the number of lost detections 1206 is less than 1, that is, 0, or if it determines that there is no tracking ID 1201 corresponding to the object tracking result of the previous frame, the process proceeds to S1305.

[0075] In S1305, the accuracy estimation unit 509 determines whether the detection reliability 1103 of the object detection result corresponding to the object tracking result is equal to or greater than a predetermined reliability threshold. For example, the accuracy estimation unit 509 determines whether the detection reliability 1103 of the object detection result of the detection ID 1101 indicated by the corresponding detection ID 1205 in the object tracking result information 1200 of the object tracking result to be processed is equal to or greater than a predetermined reliability threshold. If the accuracy estimation unit 509 determines that the detection reliability 1103 is equal to or greater than the reliability threshold, the accuracy estimation unit 509 returns the process to S1302 with the next object tracking result as the processing target. On the other hand, if the accuracy estimation unit 509 determines that the detection reliability 1103 is less than the reliability threshold, the accuracy estimation unit 509 proceeds to S1306. Note that, in S1305, an example was given in which it was determined whether the detection reliability 1103 of the object detection result is equal to or greater than a predetermined reliability threshold. However, the accuracy estimation unit 509 may also similarly determine whether the tracking reliability 1203 of the object tracking result is equal to or greater than a predetermined reliability threshold. For example, if either the detection reliability 1103 or the tracking reliability 1203 is less than a predetermined reliability threshold, the accuracy estimation unit 509 proceeds to the process of S1306.

[0076] In S1306, the accuracy estimation unit 509 calculates a first accuracy region. As part of the calculation process for the first accuracy region, the accuracy estimation unit 509 generates accuracy region information based on estimated accuracy, which indicates that the region is difficult to detect an object from within the tracking rectangle (bounding box) of the object tracking result being processed.

[0077] FIG. 15 is a diagram showing an example of precision region information 1500. As shown in FIG. The accuracy region information 1500 includes coordinates (X, Y) 1501 and accuracy region value 1502. The coordinates (X, Y) 1501 indicate X and Y coordinates. The coordinates (X, Y) 1501 store the coordinates (X, Y) of each pixel position included in the tracking rectangle (bounding box) represented by the bounding box coordinates 1204 corresponding to the object tracking result being processed. The accuracy region value 1502 stores a numerical value of estimated accuracy indicating whether the region is difficult to detect an object or not. The larger the numerical value stored in the accuracy region value 1502, the more difficult the region is to detect. The accuracy region value 1502 is calculated using the following formula (1).

[0078] Accuracy area value = α · β n Formula (1)

[0079] Here, α in formula (1) corresponds to the maximum pixel value expressed in the precision region image defined as the precision score. β is a value defined as an attenuation rate, and a value between 0 and 1 is used. n is a value corresponding to the number of lost detections 1206 in the object tracking result. Note that these values ​​may be set to any value, in addition to their definitions. After S1306, the precision estimation unit 509 proceeds to the process of S1308.

[0080] Also, when the process proceeds to S1307, the accuracy estimation unit 509 calculates a second accuracy region. As a second accuracy region calculation process, when the tracking unit 508 becomes unable to track a tracking object that it had been tracking based on the object detection result by the detection unit 507, the accuracy estimation unit 509 estimates a tracking rectangle region corresponding to the object that can no longer be tracked as a difficult-to-detect region. Furthermore, the accuracy estimation unit 509 performs an update process to set a tracking rectangle region corresponding to an object that the tracking unit 508 was tracking before the tracking unit 508 became unable to track the tracking object as a difficult-to-detect region. Specifically, the accuracy estimation unit 509 creates accuracy region information by updating a region in a past object tracking result defined by the same tracking ID 1201 as the object being processed, up until the number of lost detections 1206 becomes 1, as a difficult-to-detect region. In other words, the accuracy estimation unit 509 updates the accuracy region value 1502 calculated for a region in a past object tracking result up until the number of lost detections 1206 becomes 1. Furthermore, the accuracy region value 1502 at this time is calculated as in the following formula (2). Note that the coordinates (X, Y) 1501 in S1308, which will be described later, store the values ​​of the bounding box coordinates 1204 of the object tracking result up to the point where the number of lost detections 1206 of the tracking object with the same tracking ID 1201 chronologically preceding the object tracking result to be processed reaches the value 1. Then, after S1307, the accuracy estimation unit 509 proceeds to the processing of S1308.

[0081] Accuracy range value = α Equation (2)

[0082] When the process proceeds to S1308, the accuracy estimation unit 509 updates the accuracy region image corresponding to the object class of the object tracking result to be processed, which is stored in the storage unit 506, based on the accuracy region information created in the processes of S1306 and S1307. For example, the accuracy estimation unit 509 compares the pixel value of the accuracy region image corresponding to the coordinates (X, Y) 1501 of the accuracy region information 1500 with the accuracy region value 1502, and uses the larger value to update the pixel value of the accuracy region image corresponding to the coordinates (X, Y) 1501. Thereafter, the accuracy estimation unit 509 proceeds to process the next tracking result.

[0083] <Image generation process flow> Fig. 16 is a flowchart showing details of the image generation process executed by the image generation unit 510 in S1006 of Fig. 10. Note that the flowchart of Fig. 16 also includes the process executed by the control unit 502. First, in S1601, the image generation unit 510 acquires a background image and a precision region image for each object class of the detection target object from the storage unit 506. As described above, the precision region image may be an image, or may be region information in which coordinates and corresponding precision information values ​​are stored.

[0084] Next, in S1602, the image generating unit 510 generates a prompt. In this embodiment, the object class of the precision region image is input in the "object class" shown in prompt 1701. A set of image coordinates (X, Y) of an area on the precision region image corresponding to the object class input as the "object class" is input in the "coordinates" shown in the prompt. Furthermore, the "coordinates" shown in the prompt are calculated by determining whether or not to randomly generate an object for each precision region that differs depending on the estimated accuracy in the precision region image. For example, in an area where an object is to be generated, coordinates (X, Y) are randomly selected within that area, and the selected coordinates are input in the "coordinates" shown in prompt 1701.

[0085] Next, in S1603, the image generation unit 510 inputs the background image and precision region image acquired in S1601 and the prompt created for each object class in S1602 to the generation AI. Then, in S1604, the image generation unit 510 executes image generation processing by generation AI processing using the input background image, accuracy region image, and prompt, and generates a generated image and correct answer data.

[0086] Figures 17(b) to 17(f) are diagrams illustrating the input data to the generation AI of image generation unit 510 and the output data from the generation AI, i.e., the generated image and correct answer data generated by the generation AI. Figure 17(b) shows a background image 1710, Figure 17(c) shows an accuracy region image 1711, Figure 17(d) shows a prompt 1712, Figure 17(e) shows a generated image 1713, and Figure 17(f) shows correct answer data 1716.

[0087] In the precision region image 1711 shown in FIG. 17(c), four types of precision regions are rendered. The prompt 1712 shown in FIG. 17(d) has an "object class" of a vehicle, and is a prompt created so that an object is generated at two coordinate positions based on the precision region image of the vehicle. 17(e) is an image generated based on a background image 1710, an accuracy region image 1711, and a prompt 1712. In the generated image 1713 of FIG. 17(e), images of vehicles 1714 and 1715 are generated and positioned within the accuracy region of the accuracy region image 1711, and are superimposed on the background image 1710. The correct answer data 1716 shown in FIG. 17(f) is data in JSON format that indicates a bounding box containing information about vehicles 1714 and 1715 for the generated image 1713. In the correct answer data 1716, "type" indicates the object class, and "position" indicates the coordinate position and size on the image of the rectangle (bounding box) that surrounds the object of each object class. Furthermore, "x" and "y" indicate the coordinate position of the upper left vertex on the rectangular image, and "width" and "height" indicate the width and height on the rectangular image. Of course, the correct answer data 1716 shown in FIG. 17(f) is an example, and the present embodiment is not limited to this example.

[0088] Next, in S1605, the image generating unit 510 sends the generated image and the correct answer data to the control unit 502, and the control unit 502 stores the generated image and the correct answer data in the storage unit 506. Thereafter, in S1606, if the user or the like stops the application, the control unit 502 stops and ends the processing of the flowchart in FIG. 16, otherwise the processing of the image generation unit 510 returns to S1601.

[0089] <Accuracy area confirmation and setting screen> 18 is a diagram showing an example of an accuracy region confirmation screen and an accuracy region setting screen generated and displayed by the setting unit 505 of the server 130 according to this embodiment. Note that the accuracy region confirmation screen and the accuracy region setting screen are actually displayed by the display unit 503, but for the sake of simplicity, a description of the display by the display unit 503 will be omitted here.

[0090] 18(a) is a diagram showing an example of the configuration of an accuracy region confirmation screen 1800. In this embodiment, the user can check the accuracy region of any object by looking at the displayed accuracy region confirmation screen 1800. The accuracy region confirmation screen 1800 includes an object class name input form 1801 , a captured image display area 1802 , an accuracy image display area 1803 , an estimated accuracy bar 1804 , a setting button 1805 , and an end button 1806 . The object class name input form 1801 is an area where the user inputs the class name of any object. The captured image display area 1802 is an area where the captured image obtained from the imaging device 110 is displayed.

[0091] The accuracy image display area 1803 is an area where an accuracy region image corresponding to the object class name input by the user in the object class name input form 1801 is displayed. When the user inputs an arbitrary object class name in the object class name input form 1801, the setting unit 505 acquires an accuracy region image corresponding to the input object class name from the storage unit 506 and displays it in the accuracy image display area 1803. Note that the accuracy image display area 1803 may display an accuracy region image generated by superimposing an object existence region corresponding to the object class input by the user.

[0092] The estimated accuracy bar 1804 is a bar that indicates the level of estimated accuracy displayed in the accuracy image display area 1803, and indicates that the higher the bar is, the higher the estimated accuracy. The setting button 1805 is a button that the user presses when he or she wishes to display an accuracy region setting screen. When the setting button 1805 is pressed by the user, the setting unit 505 displays an accuracy region setting screen 1810 as shown in FIG. 18(b), for example. The end button 1806 is a button that the user presses when he or she wishes to close the accuracy region confirmation screen 1800. When the end button 1806 is pressed by the user, the setting unit 505 closes the accuracy region confirmation screen 1800 that has been displayed.

[0093] Fig. 18(b) is a diagram showing the configuration of an accuracy region setting screen 1810 that is displayed when the user presses the setting button 1805 on the accuracy region confirmation screen 1800 in Fig. 18(a). The user can set or reset a new accuracy region for the accuracy region of any object through operations on the accuracy region setting screen 1810 in Fig. 18(b).

[0094] The accuracy region setting screen 1810 includes an object class name input form 1801, a captured image display area 1802, an accuracy image display area 1803, an estimated accuracy bar 1804, a slider 1813, a setting reflect button 1814, and a setting end button 1815. The object class name input form 1801, the captured image display area 1802, the accuracy image display area 1803, and the estimated accuracy bar 1804 are the same as those in FIG. 18(a).

[0095] In the case of the accuracy area setting screen 1810 of FIG. 18(b), the user can arbitrarily set a rectangle (referred to as an accuracy area setting rectangle) representing an area where object detection is difficult on the captured image while viewing the captured image displayed in the captured image display area 1802. In the example of FIG. 18(b), an accuracy area setting rectangle 1811 is displayed as a rectangle representing an area where object detection is difficult arbitrarily set by the user. For example, when the user inputs a rectangular area on the captured image through an input operation using a mouse, a keyboard, or the like, the setting unit 505 displays the accuracy area setting rectangle 1811 corresponding to the rectangular area input by the user in the captured image display area 1802. Furthermore, when the accuracy area setting rectangle 1811 is set in the captured image display area 1802, the setting unit 505 displays the accuracy area setting rectangle 1812 in the accuracy image display area 1803 at a coordinate position corresponding to the accuracy area setting rectangle 1811 in the accuracy image display area 1803.

[0096] Furthermore, in the case of the accuracy area setting screen 1810, the setting unit 505 also displays a slider 1813 that can be slid on the estimated accuracy bar 1804 in response to an input operation by the user. The user can slide the slider 1813 to any position on the estimated accuracy bar 1804. The setting unit 505 reflects the same estimated accuracy in the accuracy area represented by the accuracy area setting rectangle 1812 so as to match the estimated accuracy indicated by the position of the slider 1813 on the estimated accuracy bar 1804. That is, the user can set the accuracy area setting rectangle 1811 in the captured image display area 1802 and further set the estimated accuracy by operating the slider 1813, thereby arbitrarily setting the estimated accuracy of the accuracy area represented by the accuracy area setting rectangle 1812.

[0097] The setting reflect button 1814 is a button that is pressed when the user desires to reflect, in the accuracy region image, the accuracy region setting rectangle 1812, which represents the accuracy region for which the user has arbitrarily set the estimated accuracy, on the accuracy region setting screen 1810. When the setting reflect button 1814 is pressed by the user, the setting unit 505 reflects the accuracy region setting rectangle 1812 in the accuracy region image. Then, the control unit 502 updates the accuracy region image by saving, in the saving unit 506, the accuracy region image in which the accuracy region setting rectangle 1812 is reflected. The setting end button 1815 is a button that is pressed when the user wishes to end the setting on the accuracy region setting screen 1810. When the setting end button 1815 is pressed, the control unit 502 closes the accuracy region setting screen 1810. After the accuracy region setting screen 1810 is closed, the setting may return to the accuracy region confirmation screen 1800, or the setting of the accuracy region confirmation screen and the accuracy region setting screen by the setting unit 505 may be ended.

[0098] As described above, in the first embodiment, based on object detection result information by the detection unit 507 and object tracking result information by the tracking unit 508, it is possible to easily create training data including images in which the object detector of the detection unit 507 has difficulty detecting objects. That is, the server 130 of this embodiment can automatically generate training data including correct answer data and generated images in which an object image corresponding to the object class is placed in an area estimated to be difficult to detect for each object class of the detection target object. Therefore, according to this embodiment, it is possible to significantly reduce the number of manual steps required for creating training data, thereby reducing the cost of creating training data.

[0099] [Second embodiment] In the first embodiment described above, an example was described in which an area where an object has not been detected or an area where detection reliability is low is estimated as an area where object detection is difficult based on the object detection result and the object tracking result. In contrast, in the second embodiment, an example is described in which an area where an object is erroneously detected as being present, even though the object to be detected is not present, is estimated as an area where detection is difficult, i.e., an area where correct detection is difficult. In the second embodiment, the system configuration, device configuration, and functional configuration are the same as those in the first embodiment, so illustrations and descriptions thereof will be omitted. Below, differences from the first embodiment will be mainly described.

[0100] <Processing flow of precision region estimation according to the second embodiment> FIG. 19 is a detailed flowchart of the accuracy region estimation process according to the second embodiment. In the second embodiment, the accuracy estimation unit 509 performs the process of the flowchart shown in FIG. 19 to estimate, as a difficult-to-detect region (a region with high estimation accuracy), even a region where an object is erroneously detected in a region where no detection target object exists. The accuracy estimation unit 509 of the second embodiment estimates a difficult-to-detect region based on at least one of the amount of movement of the tracked object, the amount of change in the aspect ratio of the rectangle (bounding box), and the size of the rectangle (bounding box) based on the object tracking result information. Note that the accuracy estimation unit 509 of the second embodiment also performs the same estimation process as described in the first embodiment, but hereinafter, only the estimation process based on the amount of movement of the tracked object, the amount of change in the aspect ratio of the rectangle, and the size of the rectangle will be described.

[0101] In the flowchart of FIG. 19 according to the second embodiment, if it is determined in S1304 that the number of lost detections 1206 is less than 1 or that there is no tracking ID 1201 corresponding to the previous object tracking result, the processing of the accuracy estimation unit 509 proceeds to S1901.

[0102] In S1901, the accuracy estimation unit 509 determines whether the amount of movement of the tracked object is within a predetermined movement amount threshold. For example, the accuracy estimation unit 509 calculates the Euclidean distance between two center coordinates calculated from the vertex coordinates of the bounding box coordinates 1204 of two tracked objects that are sequentially located in time series and have the same tracking ID 1201 in the object tracking result information 1200. The accuracy estimation unit 509 then determines whether the calculated Euclidean distance, i.e., the amount of movement of the tracked object, is within a predetermined distance threshold. If the accuracy estimation unit 509 determines that the distance is within the distance threshold as a result of this determination process, the accuracy estimation unit 509 proceeds to S1902. On the other hand, if the accuracy estimation unit 509 determines that the distance exceeds the distance threshold, the accuracy estimation unit 509 proceeds to S1306 and performs the calculation process of the first accuracy region described above.

[0103] When the process proceeds to S1902, the accuracy estimation unit 509 determines whether the amount of change in the aspect ratio of the tracking rectangles (bounding boxes) of two tracking objects that are adjacent in time series and have the same tracking ID 1201 is within a predetermined change threshold. For example, the accuracy estimation unit 509 calculates the aspect ratio of each tracking rectangle from the vertex coordinates of the bounding box coordinates 1204 of the tracking rectangles of the two tracking objects that have the same tracking ID 1201. Furthermore, the accuracy estimation unit 509 calculates the amount of change in the aspect ratio by dividing the aspect ratio of the later tracking rectangle in time series by the aspect ratio of the earlier tracking rectangle in time series. The accuracy estimation unit 509 then determines whether the amount of change in the aspect ratio is within a predetermined change threshold, and if it is determined that the amount of change is within the change threshold, the process proceeds to S1903. On the other hand, if it is determined that the amount of change exceeds the change threshold, the accuracy estimation unit 509 proceeds to S1306, where it performs the calculation process for the first accuracy region described above.

[0104] When the process proceeds to S1903, the accuracy estimation unit 509 determines whether the size of the tracking rectangle (bounding box) is within a predetermined size threshold. For example, the accuracy estimation unit 509 calculates the area of ​​the tracking rectangle from the vertex coordinates of the bounding box coordinates 1204 corresponding to the tracking rectangle, and determines whether the area is within a predetermined area threshold. If the accuracy estimation unit 509 determines that the area is within the area threshold, the process proceeds to S1305, where it performs a process of determining whether the detection reliability described above is equal to or greater than the reliability threshold. On the other hand, if it determines that the area threshold is exceeded, the accuracy estimation unit 509 proceeds to S1306, where it performs the process of calculating the first accuracy region described above.

[0105] As described above, in the second embodiment, the accuracy region estimation process is performed based on at least one of the amount of movement over time, the amount of change in the aspect ratio of the rectangle (bounding box), and the size of the tracking rectangle. As a result, according to the second embodiment, even an area where there is a possibility of false detection, such as an object being mistakenly detected in an area where the detection target object does not exist, can be estimated as an area where object detection is difficult. Therefore, according to the second embodiment, it is possible to increase the variety of training data creation that is difficult for the detection unit 507.

[0106] [Third embodiment] In the third embodiment, an example will be described in which a first precision region calculation process and a second precision region calculation process performed during a precision region estimation process are different from those of the first embodiment. In the first precision region calculation process and the second precision region calculation process according to the third embodiment, statistical information is used when calculating the precision region, making it possible to estimate a more subdivided precision region than in the first embodiment. In the third embodiment, the system configuration, device configuration, and functional configuration are the same as those of the first embodiment, so illustrations and descriptions thereof will be omitted. Below, differences from the first embodiment will be mainly described.

[0107] <Processing flow of precision region estimation according to the third embodiment> Fig. 20 is a detailed flowchart of the precision region estimation process according to the third embodiment. Fig. 21 shows an example of input data for creating a precision region image for each object class of a detection target object according to the third embodiment, and is a diagram showing precision region information calculated by the processes of S2001 to S2003 in Fig. 20.

[0108] As shown in FIG. 21 , accuracy region information 2100 according to the third embodiment is created for each object class and includes coordinates (X, Y) 2101, a total accuracy region value 2102, a total number of detections 2103, and a total estimated number of detections 2104. The coordinates (X, Y) 2101 are a set of X and Y coordinates, corresponding to the coordinates of each pixel position in the captured image and the accuracy region image. That is, if the captured image is a 1080×720 pixel image, 1080×720 records are stored in the coordinates (X, Y) 2101. The total accuracy region value 2102 stores the sum of the accuracy region values ​​calculated at the X and Y coordinate positions corresponding to the coordinates (X, Y) 2101. The total number of detections 2103 stores the number of times an object is detected at the corresponding coordinates (X, Y) 2101. The total estimated number of detections 2104 stores the number of times the corresponding coordinates (X, Y) 2101 are determined to be an area where object detection is difficult.

[0109] 20 , if it is determined in S1303 that a corresponding detection ID 1205 does not exist, the accuracy estimation unit 509 proceeds to the first accuracy region calculation process of S2001 according to this embodiment. Also, if it is determined in S1305 that the detection reliability 1103 is equal to or greater than the reliability threshold, the accuracy estimation unit 509 also proceeds to the first accuracy region calculation process of S2001. On the other hand, if it is determined in S1305 that the detection reliability 1103 is less than the reliability threshold, the accuracy estimation unit 509 proceeds to the accuracy region information update process of S2002, which will be described later. Also, if it is determined in S1304 that the detection lost count 1206 is equal to or greater than 1, the accuracy estimation unit 509 proceeds to the second accuracy region calculation process of S2003 according to this embodiment. After the processes of S2001, S2002, and S2003, the accuracy estimation unit 509 proceeds to accuracy region image update processing of S2004, which will be described later.

[0110] When the process proceeds to S2001, the accuracy estimation unit 509 executes a first accuracy region calculation process according to the third embodiment. As the first accuracy region calculation process according to the third embodiment, the accuracy estimation unit 509 calculates an accuracy region value of an area corresponding to the bounding box coordinates 1204 of the object tracking result to be processed. Then, the accuracy estimation unit 509 updates the accuracy region information 2100 using the calculated accuracy region value. Specifically, as the accuracy region information update process, the accuracy estimation unit 509 updates the record of coordinates (X, Y) 2101 corresponding to the X, Y coordinates of the bounding box coordinates 1204 of the object tracking result, in the accuracy region information 2100. In the case of S2001, the accuracy estimation unit 509 adds a value calculated by the operation of the following formula (3) to the total accuracy region value 2102, and also updates the values ​​of the total number of detections 2103 and the total estimated number of detections 2104 by adding 1 to each of them.

[0111] Accuracy area value = α · β n Formula (3)

[0112] When the process proceeds to S2002, the accuracy estimation unit 509 updates the accuracy region information. For example, as the accuracy region information update process, the accuracy estimation unit 509 updates the record of the coordinates (X, Y) 2101 corresponding to the coordinates X, Y in the bounding box coordinates 1204 of the object tracking result to be processed, in the accuracy region information 2100. In S2002, the accuracy estimation unit 509 updates the value of the total number of detections 2103 by adding 1.

[0113] When the process proceeds to S2003, the accuracy estimation unit 509 performs a second accuracy region calculation process according to the third embodiment. As the second accuracy region calculation process, the accuracy estimation unit 509 updates the accuracy region information 2100 for a region up to the point where the number of lost detections 1206 of the tracked object in the past object tracking result associated with the same tracking ID 1201 as the object tracking result to be processed becomes 1. That is, the accuracy estimation unit 509 updates the record of the coordinates (X, Y) 2101 corresponding to the coordinates X, Y of the bounding box coordinates 1204 of the object tracking result for which a series of processes are performed, so as to update the total accuracy region value 2102 calculated using the past accuracy region value. The accuracy region value in S2003 is calculated according to the following formula (4). The accuracy estimation unit 509 updates the total accuracy region value 2102 by adding the value calculated using formula (4) to it.

[0114] Accuracy domain value = α(1-β n ) Formula (4)

[0115] When the process proceeds to S2004 after the processing of S2001, S2002, or S2003 described above, the accuracy estimation unit 509 updates the pixel values ​​of the accuracy region image stored in the storage unit 506 based on the accuracy region information 2100. The accuracy estimation unit 509 updates the pixel values ​​of the X and Y coordinates of the accuracy region image corresponding to the coordinate (X, Y) 2101 of the accuracy region information 2100 using the total accuracy region value 2102 of the coordinate (X, Y) 2101, the total number of detections 2103, and the total estimated number of detections 2104 according to equation (5). Note that the pixel (X, Y) in equation (5) indicates the pixel value of the X and Y coordinate position in the accuracy region image.

[0116] Pixel (X, Y) = Total precision area value · ((Total estimated detection circuits) / (Total number of detections)) Equation (5)

[0117] As described above, in the third embodiment, in precision region estimation, the precision region is calculated using statistical information such as the total precision region value 2102, the total number of detections 2103, and the total estimated number of detections 2104. As a result, according to the third embodiment, it is possible to estimate a region that is statistically important and where object detection is difficult, based on the detection frequency.

[0118] [Fourth embodiment] In the first to third embodiments, a rule-based method for estimating an accuracy region using object detection results and object tracking results has been described. In the fourth embodiment, an example of estimating an accuracy region using an image generation model will be described. In the fourth embodiment, the system configuration, device configuration, and functional configuration are the same as those in the first to third embodiments, so illustrations and descriptions thereof will be omitted. Below, differences from the first to third embodiments will be described.

[0119] In the fourth embodiment, the accuracy estimation unit 509 executes an image generation process using object detection result information, object tracking result information, object existence region information, and a prompt (described later) as input data to generate an accuracy region image. The image generator used by the accuracy estimation unit 509 in the image generation process is, as described above, an image generation model trained by machine learning that finds patterns hidden in large amounts of data through repeated calculations. For example, the technology described in the aforementioned references can be applied as the image generation model. In addition, in the fourth embodiment, the storage unit 506 stores, in addition to the same information stored by the storage unit 506 in the previous embodiments, an image generation model used by the image generator of the accuracy estimation unit 509.

[0120] <Image generation process flow for generating precision region images> Fig. 22 is a flowchart showing the overall flow of information processing executed in each functional unit of the server 130 according to the fourth embodiment. In the flowchart of Fig. 22, the processes of S1001 to S1003, S1005, and S1006 are generally similar to those in Fig. 10, and therefore description thereof will be omitted. In the flowchart of Fig. 22, after processing S1003, the accuracy estimation unit 509 performs processes S2201 to S2203, and then the processing of the server 130 proceeds to S1005.

[0121] In S2201, the accuracy estimation unit 509 inputs (sets) the object detection result information, object tracking result information, object existence area information, and a prompt (described later) stored in the storage unit 506 to a generation AI for generating a precision area image. Fig. 23 is a diagram showing an example of a prompt 2300 input to the generation AI for generating a precision area image in S2201.

[0122] Next, in S2202, the accuracy estimation unit 509 receives the object detection result information, object tracking result information, object existence area information, and prompt as input and executes image generation processing using generation AI processing to generate an accuracy area image and correct answer data. Next, in S2203, the accuracy estimation unit 509 stores the accuracy region image and the correct answer data generated in S2202 in the storage unit 506 via the control unit 502. As described above, in the fourth embodiment, the accuracy estimation unit 509 can create an accuracy region image using an image generation model.

[0123] [Fifth embodiment] In the first to third embodiments, several methods for estimating the accuracy region have been described. In the fifth embodiment, an example will be described in which, in addition to estimating the accuracy region, processing is performed to estimate detailed parameters used when generating an image using an image generation model. In the fifth embodiment, the system configuration, device configuration, and functional configuration are the same as those of the first to third embodiments, so illustrations and descriptions thereof will be omitted. Below, differences from the first to third embodiments will be described.

[0124] <Processing flow according to the fifth embodiment> Fig. 24 is a detailed flowchart of the accuracy range estimation process performed by the accuracy estimation unit 509 according to the fifth embodiment. In the flowchart of Fig. 24, the accuracy estimation unit 509 proceeds to the process of S2401 after the processes of S1306 and S1307, and further proceeds to the process of S1308 after the process of S2041.

[0125] When the process proceeds to S2401, the accuracy estimation unit 509 estimates detailed parameters to be used in the image generation process of the image generation unit 510 for the areas where object detection is difficult (areas with high estimation accuracy) estimated in S1306 to S1308. In the case of this embodiment, the accuracy estimation unit 509 estimates at least one of, for example, "weather", "time period", and "color" as detailed parameters to be used in the image generation process of the image generation unit 510. In the case of this embodiment, the accuracy estimation unit 509 estimates each of the parameters "weather", "time period", and "color", but is not limited to this, and may estimate only one of these parameters, or may estimate two parameters, or further parameters.

[0126] Fig. 25 is a flowchart of the parameter estimation process performed in S2401 of Fig. 24. Fig. 26 is a diagram showing an example of detailed parameter information 2600 that stores parameters estimated in the processes of S2501 to S2503 of the flowchart of Fig. 25. In Fig. 26, the detailed parameter information 2600 includes coordinates (X, Y) 2601, weather parameters 2602, time period parameters 2603, and color parameters 2604. The detailed parameter information 2600 estimated and acquired by the accuracy estimation unit 509 is stored in the storage unit 506.

[0127] First, in S2501, the accuracy estimation unit 509 estimates weather parameters. For example, the accuracy estimation unit 509 estimates the weather parameters using a trained model that has been trained to estimate weather from an image using machine learning. For example, the accuracy estimation unit 509 uses an image that is the target of the accuracy region estimation process described above as input, performs parameter estimation processing using the trained model, and acquires text data indicating the weather as the parameter estimation result. Then, the accuracy estimation unit 509 adds the weather text data to the weather parameters 2602 of the coordinates (X, Y) 2601 corresponding to the region in the detailed parameter information 2600 where object detection was estimated to be difficult in the accuracy region estimation process of S1306 to S1308.

[0128] Next, in S2502, the accuracy estimation unit 509 estimates time period parameters. The accuracy estimation unit 509 classifies images for which accuracy regions are to be estimated into predefined time periods, for example, based on a time stamp assigned to the captured image when the imaging device 110 captured the image. For example, if time periods are defined as "morning" from 6:00 to 10:00, "afternoon" from 10:00 to 16:00, "evening" from 16:00 to 18:00, and "night" from 18:00 to 6:00, the accuracy estimation unit 509 acquires text data for the time period corresponding to the time stamp of the captured image. Then, the accuracy estimation unit 509 adds the text data for the time period to the time period parameter 2603 of the coordinates (X, Y) 2601 corresponding to the region in the detailed parameter information 2600 where object detection was estimated to be difficult in the accuracy region estimation process of S1306 to S1308.

[0129] Next, in S2503, the accuracy estimation unit 509 estimates color parameters. For example, the accuracy estimation unit 509 estimates color parameters from an image obtained by cutting out a rectangular bounding box from the image for which the accuracy region is to be estimated, the bounding box being estimated as a region where object detection is difficult in S1306 and S1307. In this embodiment, the accuracy estimation unit 509 performs a process of allocating the cut-out image to predefined colors based on the statistics of the color information of the cut-out image. For example, the accuracy estimation unit 509 converts the cut-out rectangular bounding box image into an HSV image representing hue, saturation, and brightness, and further calculates the most frequent H, S, and V values ​​of the pixels in the cut-out image as the statistics of the cut-out image. In this embodiment, color text data corresponding to the H, S, and V values ​​of the pixels is defined in advance, and the accuracy estimation unit 509 acquires the color text data based on the values ​​calculated as the statistics as described above. In addition to the examples described above, the statistics may also use averages or medians. Then, the accuracy estimation unit 509 adds color text data to the color parameter 2604 of the coordinates (X, Y) 2601 corresponding to the area in the detailed parameter information 2600 where object detection was estimated to be difficult in S1306 to S1307.

[0130] Fig. 27 is a flowchart of image generation processing executed by the image generation unit 510 using parameters estimated by the accuracy estimation unit 509 as described above in the fifth embodiment. In the flowchart of Fig. 27, steps S1603 to S1606 are the same as those in the flowchart of Fig. 16. In the flowchart of Fig. 27, the image generation unit 510 performs the processes of S2701 and S2702, and then proceeds to the process of S1603.

[0131] First, in S2701, the image generation unit 510 acquires the background image, the precision region image, and the detailed parameter information stored in the storage unit 506. Next, in S2702, the image generation unit 510 generates a prompt, which will be described later. A prompt 2801 in Fig. 28(a) and a prompt 2802 in Fig. 28(b) are diagrams showing examples of prompts generated by the image generation unit 510.

[0132] The object class of the precision region image acquired from the storage unit 506 is input into the "object class" prompt 2801. A set of coordinates (X, Y) of the precision region image corresponding to the object class input into the "object class" prompt 2801 is input into the "coordinates" prompt 2801. The "coordinates" prompt 2801 are calculated based on the result of determining whether or not to randomly generate an object for each precision region that differs depending on the estimated accuracy within the precision region image. For example, in a region where an object is to be generated, coordinates (X, Y) are randomly selected within that region, and the selected coordinates are input into the "coordinates" prompt 2801. A value randomly selected from the color parameters of the detailed parameter information of the randomly selected coordinate position is input into the "color" prompt 2801. Values ​​selected from the detailed parameter information of the randomly selected coordinate position so as to cover all of the corresponding weather parameters and time zone parameters are input into the "weather" and "time zone" prompt 2801. That is, if multiple values ​​are stored in the "weather parameter" and "time period parameter" at a randomly selected coordinate location, prompts are created to cover all of those patterns.

[0133] Prompt 2802 in Figure 28(b) is created so that the "object class" is a vehicle, an object is generated at a single coordinate position based on the precision area image of the vehicle, and the "color" is black, the "weather" is sunny, and the "time zone" is evening based on the detailed parameter information of that coordinate position.

[0134] As described above, in the fifth embodiment, when an area where object detection is difficult is estimated, the accuracy estimation unit 509 performs processing to estimate detailed parameters such as weather, time of day, and object color information. Then, in the fifth embodiment, the image generation unit 510 performs processing to generate an image using these detailed parameters, which makes it possible to more effectively create learning data that is difficult for the object detector of the detection unit 507.

[0135] The present invention can also be realized by providing a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely examples of specific embodiments for implementing the present invention, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0136] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) a detection means for detecting an object from a captured image; a tracking means for tracking the object on time-series captured images based on the result of the object detection by the detection means; an estimation means for estimating, from the captured image, an area in which it is difficult for the detection means to detect the object for each type of object, based on a detection result of the object by the detection means and a tracking result of the object by the tracking means; a generating means for generating an image in which a predetermined object image corresponding to the type of the object is superimposed on a predetermined background image based on the estimation result by the estimating means; An information processing device comprising: (Configuration 2) 2. The information processing apparatus according to configuration 1, wherein the estimation means performs the estimation for an area of ​​the captured image that is defined as containing the object. (Configuration 3) The information processing device according to configuration 1 or 2, wherein the estimation means acquires an estimation accuracy of an area where it is difficult to detect the object for each type of object, and determines an area where it is difficult for the detection means to detect the object based on the estimation accuracy. (Configuration 4) The information processing device according to configuration 3, wherein the estimation means increases the estimation accuracy of a second region corresponding to the object that has not been detected by the detection means and that has been tracked by the tracking means, compared to a first region corresponding to the object that has been detected by the detection means from the captured image, based on the detection result of the object and the tracking result of the object. (Configuration 5) The information processing device according to configuration 4, wherein the estimation means increases the estimation accuracy of a third area in which neither the object is detected by the detection means nor the object is tracked by the tracking means, compared to the second area. (Configuration 6) 4. The information processing device according to configuration 3, wherein the generating means generates the superimposed image by preferentially arranging the predetermined object image in the predetermined background image starting from an area with high estimation accuracy. (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the estimation means estimates an area in which it is difficult for the detection means to detect the object, based on at least one of the reliability of the detection by the detection means and the reliability of the tracking by the tracking means. (Configuration 8) The information processing device according to configuration 7, wherein the estimation means estimates an area where at least one of the reliability of the detection by the detection means and the reliability of the tracking by the tracking means is less than a predetermined reliability threshold as an area where it is difficult for the detection means to detect the object. (Configuration 9) 9. The information processing device according to any one of configurations 1 to 8, further comprising a setting unit that sets an area set by a user in the captured image as an area where it is difficult for the object to be detected by the detection unit. (Configuration 10) 10. The information processing device according to any one of configurations 1 to 9, wherein the estimation means estimates an area in which it is difficult for the detection means to detect the object, based on the amount of movement of the object tracked by the tracking means. (Configuration 11) 11. The information processing device according to configuration 10, wherein the estimation means estimates an area where the movement amount of the object exceeds a predetermined movement amount as an area where it is difficult for the detection means to detect the object. (Configuration 12) 12. The information processing device according to any one of configurations 1 to 11, wherein the estimation means estimates an area in which it is difficult for the detection means to detect the object, based on a first aspect ratio of an area corresponding to the object tracked earlier in time series by the tracking means and a second aspect ratio of an area corresponding to the object tracked later in time series by the tracking means. (Configuration 13) 13. The information processing device according to claim 12, wherein the estimation means estimates an area in which it is difficult for the detection means to detect the object, based on an amount of change in the second aspect ratio relative to the first aspect ratio. (Configuration 14) The information processing device according to configuration 13, wherein the estimation means calculates the amount of change by dividing the first aspect ratio by the second aspect ratio, and estimates an area where the amount of change exceeds a predetermined change amount threshold as an area where it is difficult for the detection means to detect the object. (Configuration 15) 15. The information processing device according to any one of configurations 1 to 14, wherein the estimation means estimates an area in which it is difficult for the detection means to detect the object, based on the size of an area corresponding to the object tracked by the tracking means. (Configuration 16) 16. The information processing apparatus according to configuration 15, wherein the estimation means estimates an area where the size exceeds a predetermined size threshold as an area where it is difficult for the detection means to detect the object. (Configuration 17) 17. The information processing device according to any one of configurations 1 to 16, wherein when the tracking means is no longer able to track the object that the tracking means was tracking based on the detection result of the object by the detection means, the estimation means estimates an area corresponding to the object that has become unable to be tracked as an area where it is difficult for the detection means to detect the object. (Configuration 18) 18. The information processing device according to configuration 17, wherein the estimation means estimates an area corresponding to the object that the tracking means was tracking before the object became unable to be tracked as an area where it is difficult for the detection means to detect the object. (Configuration 19) The information processing device according to any one of configurations 3 to 6, wherein the estimation means determines the estimation accuracy of the area where it is difficult for the detection means to detect the object, based on the number of times the object is detected by the detection means and the number of times the area is estimated as being difficult to detect the object, for each pixel coordinate of the captured image. (Configuration 20) The information processing device according to any one of configurations 3 to 6, wherein the estimation means determines the estimation accuracy for each type of object based on a detection result of the object by the detection means, a tracking result of the object by the tracking means, and a prompt created for each type of object. (Configuration 21) The information processing device described in any one of configurations 1 to 20, wherein the estimation means acquires at least one of the weather, time of day, and color of the area when the captured image in which the area where the object is difficult to detect was estimated as a parameter to be used by the generation means to generate the image. (Configuration 22) The information processing device according to configuration 21, characterized in that the generation means generates an image in which the predetermined object image is superimposed on the background image, reflecting at least one parameter of the weather, the time of day, and the color of the area. (Method 1) a detection step of detecting an object from the captured image; a tracking step of tracking the object on time-series captured images based on a result of the object detection by the detection step; an estimation step of estimating, from the captured image, an area in which it is difficult to detect the object in the detection step, for each type of object, based on the detection result of the object in the detection step and the tracking result of the object in the tracking step; a generating step of generating an image in which a predetermined object image corresponding to the type of the object is superimposed on a predetermined background image based on the estimation result of the estimating step; An information processing method comprising: (Program 1) A program that causes a computer to function as the information processing device according to any one of configurations 1 to 22. [Explanation of symbols]

[0137] 110: imaging device, 120: network, 130: server, 501: communication unit, 502: control unit, 503: display unit, 504: operation unit, setting unit: 505, storage unit: 506, detection unit: 507, tracking unit: 508, accuracy estimation unit: 509, image generation unit: 510

Claims

1. a detection means for detecting an object from a captured image; a tracking means for tracking the object on time-series captured images based on the result of the object detection by the detection means; an estimation means for estimating, from the captured image, an area in which it is difficult for the detection means to detect the object for each type of object, based on a detection result of the object by the detection means and a tracking result of the object by the tracking means; a generating means for generating an image in which a predetermined object image corresponding to the type of the object is superimposed on a predetermined background image based on the estimation result by the estimating means; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the estimation means performs the estimation for a region of the captured image that is defined as containing the object.

3. 3. The information processing device according to claim 1, wherein the estimation means acquires an estimation accuracy of an area in which it is difficult to detect the object for each type of object, and determines an area in which it is difficult for the detection means to detect the object based on the estimation accuracy.

4. 4. The information processing device according to claim 3, wherein the estimation means increases the estimation accuracy of a second region corresponding to the object detected by the detection means from the captured image by the object detection means, compared to a first region corresponding to the object detected by the detection means, where the object is not detected by the detection means and is tracked by the tracking means, based on the object detection result and the object tracking result.

5. 5. The information processing device according to claim 4, wherein the estimation means increases the estimation accuracy of a third area in which neither the object is detected by the detection means nor the object is tracked by the tracking means, compared to the second area.

6. 4. The information processing apparatus according to claim 3, wherein the generating means generates the superimposed image by arranging the predetermined object image in the predetermined background image, preferentially starting from an area with high estimation accuracy.

7. 2. The information processing apparatus according to claim 1, wherein the estimation means estimates an area in which it is difficult for the detection means to detect the object, based on at least one of the reliability of the detection by the detection means and the reliability of the tracking by the tracking means.

8. The information processing device according to claim 7, characterized in that the estimation means estimates an area in which at least one of the reliability of the detection by the detection means and the reliability of the tracking by the tracking means is less than a predetermined reliability threshold as an area in which it is difficult for the detection means to detect the object.

9. 2. The information processing apparatus according to claim 1, further comprising a setting unit that sets an area set by a user in the captured image as an area in which it is difficult for the object to be detected by the detection unit.

10. 2. The information processing apparatus according to claim 1, wherein the estimation means estimates an area where it is difficult for the detection means to detect the object based on the amount of movement of the object tracked by the tracking means.

11. 11. The information processing apparatus according to claim 10, wherein the estimation means estimates an area where the movement amount of the object exceeds a predetermined movement amount as an area where it is difficult for the detection means to detect the object.

12. 2. The information processing device according to claim 1, wherein the estimation means estimates an area in which it is difficult for the detection means to detect the object, based on a first aspect ratio of an area corresponding to the object tracked earlier in the time series by the tracking means and a second aspect ratio of an area corresponding to the object tracked later in the time series by the tracking means.

13. 13. The information processing apparatus according to claim 12, wherein the estimation means estimates an area in which it is difficult for the detection means to detect the object, based on an amount of change in the second aspect ratio relative to the first aspect ratio.

14. The information processing device according to claim 13, characterized in that the estimation means calculates the amount of change by dividing the first aspect ratio by the second aspect ratio, and estimates an area where the amount of change exceeds a predetermined change amount threshold as an area where it is difficult for the detection means to detect the object.

15. 2. The information processing apparatus according to claim 1, wherein the estimation means estimates an area in which it is difficult for the detection means to detect the object based on the size of an area corresponding to the object tracked by the tracking means.

16. 16. The information processing apparatus according to claim 15, wherein the estimation means estimates an area where the size exceeds a predetermined size threshold as an area where it is difficult for the detection means to detect the object.

17. 2. The information processing device according to claim 1, wherein, when the tracking means is no longer able to track the object that the tracking means was tracking based on the detection result of the object by the detection means, the estimation means estimates an area corresponding to the object that has become unable to be tracked as an area in which it is difficult for the detection means to detect the object.

18. 18. The information processing apparatus according to claim 17, wherein the estimation means estimates, as a region where it is difficult for the detection means to detect the object, a region corresponding to the object that the tracking means had been tracking before the object became unable to be tracked.

19. The information processing device according to claim 3, characterized in that the estimation means determines the estimation accuracy of the area in which it is difficult for the detection means to detect the object based on the number of times the object is detected by the detection means and the number of times the area is estimated as being difficult to detect the object, for each pixel coordinate of the captured image.

20. 4. The information processing device according to claim 3, wherein the estimation means determines the estimation accuracy for each type of object based on the detection result of the object by the detection means, the tracking result of the object by the tracking means, and a prompt created for each type of object.

21. The information processing device according to claim 1, characterized in that the estimation means acquires at least one of the weather, time of day, and color of the area when the captured image in which the area where the object is difficult to detect was captured as a parameter to be used by the generation means to generate the image.

22. 22. The information processing device according to claim 21, wherein the generating means generates an image in which the predetermined object image is superimposed on the background image, reflecting at least one parameter of the weather, the time of day, and the color of the area.

23. a detection step of detecting an object from the captured image; a tracking step of tracking the object on time-series captured images based on a result of the object detection by the detection step; an estimation step of estimating, from the captured image, an area in which it is difficult to detect the object in the detection step, for each type of object, based on the detection result of the object in the detection step and the tracking result of the object in the tracking step; a generating step of generating an image in which a predetermined object image corresponding to the type of the object is superimposed on a predetermined background image based on the estimation result of the estimating step; An information processing method comprising:

24. Computer, a detection means for detecting an object from a captured image; a tracking means for tracking the object on time-series captured images based on the result of the object detection by the detection means; an estimation means for estimating, from the captured image, an area in which it is difficult for the detection means to detect the object for each type of object, based on a detection result of the object by the detection means and a tracking result of the object by the tracking means; a generating means for generating an image in which a predetermined object image corresponding to the type of the object is superimposed on a predetermined background image based on the estimation result by the estimating means; A program that causes the device to function as an information processing device having the above.

Images