Image processing device, image processing method, and image processing program

Abstract

This invention provides an image processing device, an image processing method, and an image processing program that can reduce the workload associated with generating a large amount of training data. [Solution] The image processing device comprises a storage unit that stores a captured image of a target area, a correspondence between each position on the captured image and the distance on the target area indicated by the unit pixel at each position, and a control unit. The control unit performs a reception process that receives a specification of a first size at a first position of the object image to be placed on the captured image, a determination process that determines a second size of the object image at a second position different from the first position on the captured image based on the first size and the correspondence stored in the storage unit, and a generation process that places the object image of the first size at the first position on the captured image and generates a composite image in which the object image of the second size is placed at the second position on the captured image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image processing program.

Background Art

[0002] In recent years, machine learning has been used in image processing. Since a large number of learning data are used in machine learning, a system for generating such learning data has been proposed (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When performing detection of fallen objects on a road using machine learning, for example, learning data obtained by synthesizing fallen object images of the fallen objects to be detected with road images of the road taken under various conditions is used. The size of the fallen object image varies depending on the position where the fallen object image is placed in the road image. Therefore, for each of a large number of road images taken under various conditions, generating learning data in which fallen object images of appropriate sizes are placed at various positions is a task that requires a great deal of labor. Such a problem is not limited to the case of applying machine learning to the detection of fallen objects on a road, but is common to the case of generating learning data in which object images to be detected are superimposed on images taken under various conditions.

[0005] One aspect of the disclosed technology aims to provide an image processing apparatus, an image processing method, and an image processing program capable of reducing the workload related to generating a large number of learning data.

Means for Solving the Problems

[0006] One aspect of the disclosed technology is exemplified by the following image processing apparatus. The image processing apparatus comprises a storage unit that stores a captured image of a target area, a correspondence between each position on the captured image and the distance on the target area indicated by a unit pixel at each of the positions, and a control unit. The control unit performs a reception process that receives a specification of a first size at a first position of the object image to be placed on the captured image, a determination process that determines a second size of the object image at a second position different from the first position on the captured image based on the first size and the correspondence stored in the storage unit, and a generation process that places the object image of the first size at the first position on the captured image and generates a composite image in which the object image of the second size is placed at the second position on the captured image.

[0007] According to this image processing device, the size of the second object image to be placed at the second position is determined based on the size of the first object image and the storage unit. Therefore, with this image processing device, it is not necessary to specify both the position and size for each of the multiple object images, thus reducing the workload involved in generating a large amount of training data.

[0008] The image processing device may further have the following features: The reception process may include a process for receiving the specification of the second position. An image processing device having such features can generate the composite image in which the second object image is placed at the operator's desired position.

[0009] The image processing device may further have the following features: The reception process includes a process for determining the second position within a predetermined area in the captured image. By further having such features, the image processing device does not need to specify the position of the second object image by the operator. Therefore, the workload related to generating a large amount of training data can be further reduced.

[0010] The image processing device may further have the following features. The storage unit stores a plurality of captured images having the same field of view, focal length, and shooting direction. The generation process includes: generating a first composite image in which the object image of a first size is placed at a first position on a first captured image and the object image of a second size is placed at a second position on the first captured image; and generating a second composite image in which the object image of a first size is placed at a first position on a second captured image different from the first captured image and the object image of a second size is placed at a second position on the second captured image. With an image processing device having such features, if the position and size of the falling object image are determined for the first composite image, the second composite image can be generated without specifying the position and size of the falling object image. Therefore, the workload related to generating a large amount of training data can be further reduced.

[0011] The technology being disclosed can also be understood from the perspective of image processing methods and image processing programs. [Effects of the Invention]

[0012] According to the disclosed technology, the workload involved in generating large amounts of training data can be reduced. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 shows an example of a falling object detection system according to an embodiment. [Figure 2] Figure 2 shows an example of the hardware configuration of the detection device according to this embodiment. [Figure 3] Figure 3 shows an example of a processing block of a detection device according to an embodiment. [Figure 4] Figure 4 is a diagram showing an example of a background image according to the embodiment. [Figure 5] Figure 5 shows an example of a background image management table according to the embodiment. [Figure 6] FIG. 6 is a drawing showing an example of an image of a falling object according to an embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a falling object management table according to an embodiment. [Figure 8] FIG. 8 is a diagram for explaining that in an embodiment, the distance in the width direction of the road corresponding to one pixel varies depending on the position on the image. [Figure 9] FIG. 9 is a diagram showing an example of a distance map according to an embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a falling object designation screen output by a reception unit in an embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a composite image according to an embodiment. [Figure 12] FIG. 12 is a diagram showing an example of a processing flow of a detection device according to an embodiment.

Embodiments for Carrying Out the Invention

[0014] <Application Example> An application example of the present invention will be described. An application example of the present invention is, for example, the detection device 2 illustrated in FIG. 1. The detection device 2 inputs a captured image of the road 500 captured by the camera 1 into a learning model 221 (see FIG. 3) to detect falling objects on the road 500. Here, the falling object is not necessarily limited to an object. The falling object may be a living thing exemplified by a human and an animal. Here, the learning model 221 constructs a composite image in which a falling object image capturing the falling object is superimposed on the captured image of the road 500 as learning data. In order to accurately detect falling objects on the road 500, it is preferable to use a large number of learning data. However, designating the position and size of the falling object for each captured image and preparing a large number of composite images will impose an enormous workload on the operator.

[0015] Therefore, in this application example, a distance map 263 (see FIG. 9) in which the distances on the road 500 to be photographed are associated with each unit pixel (for example, 1 pixel) at each position on the photographed image is stored in advance in the auxiliary storage unit 203 (see FIG. 2) of the detection device 2. Then, when the position and size of a certain falling object (the first falling object) are specified in the detection device 2, for a second falling object different from the first falling object, only the position is accepted for specification. Then, the detection device 2 determines the size of the second falling object by referring to the distance map 263 based on the size of the first falling object.

[0016] According to this application example, it is not necessary to specify both the position and the size for each of the plurality of falling objects. Therefore, according to this application example, the workload related to generating a large number of learning data can be reduced.

[0017] <Embodiment> Hereinafter, the embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing an example of a falling object detection system 100 according to the embodiment. The falling object detection system 100 is a system that detects an object that has fallen on the road 500. The falling object detection system 100 includes a camera 1, a detection device 2, and a display device 3. The camera 1 and the detection device 2 are connected by a computer network N1. The detection device 2 and the display device 3 are connected by a connection cable L1. The road 500 is, for example, an expressway for automobiles or an express national highway for automobiles. The road 500 may be a general national highway, a prefectural road, a municipal road, or a private road.

[0018] Camera 1 is a video camera that films the road 500 to be monitored. Camera 1 may be a network camera that transmits captured images via a computer network N1, for example. Camera 1 is a digital video camera that employs a Charge Coupled Device (CCD) or Complementary Metal-Oxide-Semiconductor (CMOS) as its image sensor. The video data captured by Camera 1 is transmitted to the detection device 2 via the computer network N1. The frame rate of the video data captured by Camera 1 is, for example, 1 fps.

[0019] Detection device 2 is an information processing device that detects fallen objects on road 500 based on video acquired from camera 1 via computer network N1. Detection device 2 performs fallen object detection using, for example, a learning model constructed by machine learning. The learning model of detection device 2 uses a composite image as training data, which is created by superimposing images of the fallen objects to be detected onto road images previously taken by camera 1. When detection device 2 detects fallen objects on road 500, it outputs the video of the detected objects to display device 3 via connecting cable L1.

[0020] Display device 3 is a display monitored by traffic monitor K1. Display device 3 is, for example, a Liquid Crystal Display (LCD), Plasma Display Panel (PDP), inorganic electroluminescence (EL) panel, or organic EL panel. Display device 3 displays information output by detection device 2.

[0021] Figure 2 shows an example of the hardware configuration of the detection device 2 according to the embodiment. The detection device 2 comprises a Central Processing Unit (CPU) 201, a main memory unit 202, an auxiliary memory unit 203, a communication unit 204, a connection terminal 205, and a bus B1. U201, main memory unit 202, auxiliary memory unit 203, communication unit 204, and connection terminal 205 are interconnected by bus B1.

[0022] The CPU201 is also called a microprocessor unit (MPU) or processor. The CPU201 is not limited to a single processor and may be in a multiprocessor configuration. Furthermore, a single CPU201 connected via a single socket may have a multicore configuration. At least some of the processing performed by the CPU201 may be performed by other processors, such as dedicated processors like Digital Signal Processors (DSPs), Graphics Processing Units (GPUs), numerical processors, vector processors, and image processing processors. Also, at least some of the processing performed by the CPU201 may be performed by integrated circuits (ICs) or other digital circuits. Furthermore, at least some of the CPU201 may include analog circuits. Integrated circuits include Large Scale Integrated Circuits (LSIs), Application Specific Integrated Circuits (ASICs), and Programmable Logic Devices (PLDs). PLDs include, for example, Field-Programmable Gate Arrays (FPGAs). The CPU201 may be a combination of a processor and integrated circuits. The combination is called, for example, a microcontroller unit (MCU), system-on-a-chip (SoC), system LSI, or chipset. In the detection device 2, the CPU 201 loads the program stored in the auxiliary storage unit 203 into the work area of ​​the main storage unit 202 and controls peripheral devices through program execution. This allows the detection device 2 to perform processing that matches a predetermined purpose. The main storage unit 202 and the auxiliary storage unit 203 are recording media that can be read by the detection device 2.

[0023] The main memory unit 202 is exemplified as a memory unit that is directly accessed by the CPU 201. The main memory unit 202 includes Random Access Memory (RAM) and Read Only Memory (ROM).

[0024] The auxiliary storage unit 203 stores various programs and data on a recording medium in a read-write manner. The auxiliary storage unit 203 is also called an external storage device. Multiple background models may be stored in the auxiliary storage unit 203. A background model is, for example, an image that shows only the background of a video of road 500 taken by camera 1. For example, a background model is generated based on a video of road 500 taken by camera 1 when no vehicles are moving on it. A background model is, for example, a grayscale image. In addition, multiple background models taken under different shooting conditions are stored in the auxiliary storage unit 203. Shooting conditions include the time of shooting, the weather at the time of shooting, the season in which the video was taken, etc.

[0025] Furthermore, the auxiliary storage unit 203 stores the operating system (OS), various programs, various tables, etc. The OS includes a communication interface program that handles data exchange with external devices connected via the communication unit 204. External devices include, for example, other information processing devices and external storage devices connected via a computer network. The auxiliary storage unit 203 may also be, for example, part of a cloud system, which is a group of computers on a network.

[0026] The auxiliary storage unit 203 is, for example, an Erasable Programmable ROM (EPROM), a Solid State Drive (SSD), a Hard Disk Drive (HDD), etc. The auxiliary storage unit 203 is also, for example, a Compact Disc (CD) drive, a Digital Versatile Disc (DVD) drive, a Blu-ray® Disc (BD) drive, etc. Furthermore, the auxiliary storage unit 203 is It may be provided by Network Attached Storage (NAS) or Storage Area Network (SAN).

[0027] The communication unit 204 is, for example, an interface with the computer network N1. The communication unit 204 communicates with external devices such as the camera 1 via the computer network N1.

[0028] Connection terminal 205 is the connection terminal to which the connection cable L1 is connected. When the detection device 2 detects a falling object, it outputs an alarm to the display device 3 via the connection cable L1.

[0029] <Processing block of detection device 2> Figure 3 shows an example of a processing block of the detection device 2 according to an embodiment. The detection device 2 includes an acquisition unit 21, an extraction unit 22, a reception unit 23, a determination unit 24, a generation unit 25, and a management database (indicated as "management DB" in the figure) 26. The detection device 2 performs processing as each of its respective units, such as the acquisition unit 21, extraction unit 22, reception unit 23, determination unit 24, generation unit 25, and management database 26, by having the CPU 201 execute a computer program that has been expanded in executable form in the main memory unit 202.

[0030] The management database 26 is a database that stores various information related to the detection of fallen objects by the detection device 2. The management database 26 is built in, for example, the auxiliary storage unit 203. The management database 26 stores, for example, frame images (also referred to as "background images") of the road 500 without vehicles or fallen objects, taken by the camera 1 under various conditions. Figure 4 is a diagram showing an example of a background image 11 according to the embodiment. As described above, the background image 11 is an image of the road 500 without vehicles or fallen objects. Multiple background images 11 taken under various conditions (weather, season, time of day, etc.) are managed in the management database 26.

[0031] Figure 5 shows an example of a background image management table 261 according to an embodiment. The background image management table 261 has the items "Condition" and "Background Image ID". The "Condition" stores the shooting conditions when the background image 11 was taken. In the example in Figure 5, the "Condition" has the items "Season", "Time of Day", and "Weather". The "Season", "Time of Day", and "Weather" items store information indicating the season, time of day, and weather when the background image 11 was taken. The "Background Image ID" stores an ID (background image ID) that uniquely identifies the background image taken by the camera 1. The "Background Image ID" may, for example, store the path name of the background image in place of the background image ID, or together with the background image ID. The background image management table 261 associates the shooting conditions under which the background image 11 was taken with the background image 11.

[0032] Furthermore, the management database 26 also manages images of fallen objects (also referred to as "fallen object images") captured by camera 1 or other imaging devices including cameras. Fallen object images are images of various fallen objects that may occur on the road 500, captured by the imaging device. Examples of fallen objects that may occur on the road 500 include cardboard boxes, tires, empty cans, empty bottles, etc. Figure 6 is a diagram showing an example of a fallen object image 12 according to the embodiment. In the example of Figure 6, a fallen object image 12 in which a cardboard box is captured as a fallen object is shown as an example. A fallen object image 12 is, for example, an image in which the margin around the captured fallen object has been removed from an image of a fallen object captured by the imaging device. As described above, the fallen object images 12 are managed by the management database 26.

[0033] Figure 7 shows an example of a falling object management table 262 according to an embodiment. The falling object management table 262 has the items "Type" and "Falling Object ID". In "Type", the falling object Information indicating the type of object is stored. In the example in Figure 7, "tire," "cardboard box," "empty bottle," and "empty can" are listed as "types" of fallen objects. The "Fallen Object ID" stores, for example, an ID (Fallen Object ID) that uniquely identifies the image of the fallen object captured by camera 1. The "Fallen Object Image ID" may store, for example, the path name of the background image in place of the Fallen Object ID, or together with the Fallen Object ID.

[0034] By the way, in the background image 11 captured by camera 1, for example, the distance in the width direction of the road 500 corresponding to one pixel differs depending on the position on the image. Figure 8 is a diagram illustrating, in an embodiment, that the distance in the width direction of the road 500 corresponding to one pixel differs depending on the position on the image. In Figure 8, a dotted line A1 set on the near side of the road 500 and a dotted line A2 set on the far side of the road 500 are shown as examples. The direction from left to right of the road 500 (width direction) is defined as the X direction, and the direction from the near side to the far side of the road 500 (up and down direction) is defined as the Y direction. In the background image 11, the road 500 is shown larger at positions with a smaller Y coordinate than at positions with a larger Y coordinate. Therefore, the distance in the X direction of the actual road 500 shown by one pixel on the background image 11 varies according to the Y coordinate. For example, in Figure 8, comparing pixel P1 on dotted line A1 and pixel P2 on dotted line A2, the distance in the X direction on the actual road 500 indicated by the width W1 of pixel P1 is greater than the distance in the X direction on the actual road 500 indicated by the width W2 of pixel P2.

[0035] Therefore, the management database 26 also manages the length per pixel in the background image 11. Figure 9 is a diagram showing an example of a distance map 263 according to the embodiment. The distance map 263 has items for "Y coordinate" and "distance". The "Y coordinate" stores information indicating the Y coordinate on the background image 11. The "distance" stores information indicating the distance on the road 500 indicated by the width of one pixel. The units of the distance information can be, for example, "mm (millimeters)", "cm (centimeters)", "m (meters)", etc. In the example of Figure 9, the unit of "distance" is, for example, "cm (centimeters)". The distance per pixel in the distance map 263 may be determined, for example, based on the spacing of cones placed on the road 500 for distance measurement. Alternatively, the distance per pixel in the distance map 263 may be determined, for example, based on the design information of the road 500. Furthermore, the distance per pixel in the distance map 263 may be determined, for example, by detecting the white lines provided on the road 500 using image processing, and based on the interval between the detected white lines and the width of the road 500. The width of the road 500 can be obtained, for example, from the design information of the road 500.

[0036] In the example in Figure 9, the distance represented by the width of one pixel is associated with the Y coordinate, but the distance map 263 is not limited to this configuration. In the distance map 263, for example, the distance represented by the width and height of one pixel may be associated with a combination of the X and Y coordinates. Here, the distance per pixel will differ if the position, field of view, focal length, etc., of the camera 1 are different. Therefore, if multiple cameras 1 are arranged in the falling object detection system 100, the distance map 263 should be prepared to correspond to each camera 1. In the example in Figure 9, the distance represented by the width of one pixel is shown with a specific numerical value for the Y coordinate, but the distance represented by the width of one pixel for the Y coordinate may also be shown by multiplying a predetermined value by a coefficient, for example. For example, the distance corresponding to the Y coordinate "1" in the distance map 263 may be expressed as "10 × α" by multiplying the distance "10" corresponding to the Y coordinate "1" by a coefficient α (where α is a predetermined real number).

[0037] Alternatively, instead of the distance map 263, a formula for calculating the distance represented by the width of one pixel relative to the Y coordinate (hereinafter also referred to as the "distance calculation formula") may be stored in the auxiliary storage unit 203. Such a formula may be determined, for example, by determining the distance per pixel at each Y coordinate based on the spacing of cones placed on the road 500 for distance measurement, and showing the relationship between the determined distance per pixel and the Y coordinate. The distance calculation formula may be, for example, L for distance and Y for the value of the Y coordinate. The coefficient can be set to β (where β is a predetermined real number), and for example, it can be expressed as "L = β × Y". Note that the distance calculation formula is not limited to a linear formula, but may be a quadratic or higher formula, and a correction value may be added. The distance map 263 and the distance calculation formula are examples of "the correspondence between each position on the captured image and the distance on the target area indicated by the unit pixel at each of the said positions".

[0038] Returning to Figure 3, the acquisition unit 21 and the extraction unit 22 are processing blocks used for detecting fallen objects on the road 500. The acquisition unit 21 acquires frame images of the road 500 captured by the camera 1. The extraction unit 22 detects fallen objects in the frame images acquired by the acquisition unit 21. The extraction unit 22 includes a learning model 221. The learning model 221 is a model constructed using composite images as training data, in which images of fallen objects to be detected are superimposed on road images of the road 500 captured by the camera 1 under various conditions. The extraction unit 22 performs fallen object detection on the road 500 by inputting the frame images into the learning model 221, for example.

[0039] The reception unit 23, the determination unit 24, and the generation unit 25 are processing blocks used, for example, to generate training data for the learning model 221. The reception unit 23, for example, outputs the background image 11 to a display such as the display device 3 to receive the specification of the type, position, and size of the falling object.

[0040] Figure 10 shows an example of a falling object specification screen 231 output by the reception unit 23 in this embodiment. The falling object specification screen 231 accepts the specification of the position and size of a falling object on the background image 11 captured by the camera 1. The falling object specification screen 231 includes an image display area 2311, a falling object selection menu 2312, and a rectangular frame 2313. The background image 11 captured by the camera 1 is displayed in the image display area 2311.

[0041] The falling object selection menu 2312 is a menu for selecting the type of falling object to be placed on the road 500 in the background image 11. The types of falling objects displayed in the falling object selection menu 2312 are obtained, for example, from the falling object management table 262. For example, an operator generating training data uses the falling object selection menu 2312 on the falling object specification screen 231 to specify the type of falling object. In the example in Figure 10, "cardboard box" is specified as the type of falling object. The operator then simply needs to make the rectangular frame 2313 the desired size and place it in the desired position within the image display area 2311. If the operator wants to place other falling objects, they can specify their positions using an input device such as a mouse, as shown at point P11 in Figure 10.

[0042] The determination unit 24 obtains the position of the image display area 2311 on the background image 11 (for example, the position of the lower left corner of the image display area 2311). The determination unit 24 then obtains the distance indicated by the width of one pixel corresponding to the obtained position from the distance map 263. The determination unit 24 then obtains the number of pixels for the width W11 and the number of pixels for the height H11 of the rectangular frame 2313. Based on the number of pixels for the width W11 and the distance indicated by the width of one pixel, the determination unit 24 calculates the distance (length) of the width W11 of the rectangular frame 2313. The determination unit 24 also calculates the ratio of height H11 to width W11 based on the number of pixels for width W11 and the number of pixels for height H11.

[0043] The determination unit 24 obtains the Y coordinate of the specified point P11. The determination unit 24 then obtains the distance indicated by the width of one pixel corresponding to the obtained Y coordinate from the distance map 263. The determination unit 24 determines the number of pixels for the width of the rectangular frame 2313 when the rectangular frame 2313 is placed on point P11 by dividing the distance W11 of the width of the rectangular frame 2313 by the distance indicated by the width of point P11. The determination unit 24 also determines the number of pixels for the height of the rectangular frame 2313 when the rectangular frame 2313 is placed on point P11, based on the determined number of pixels for the width of the rectangular frame 2313 and the ratio of the height H11 to the width W11. Once the number of pixels for the width and the number of pixels for the height are determined, the size of the rectangular frame 2313 when placed on point P11 is determined. The determination unit 24 also uses the distance map 263. If a distance calculation formula is stored in the auxiliary storage unit 203 instead of 3, the distance indicated by the width of one pixel may be calculated using the distance calculation formula.

[0044] The generation unit 25 generates a composite image by superimposing images of the fallen object received by the reception unit 23, placed at the position received by the reception unit 23, and at the size determined by the determination unit 24. Figure 11 shows an example of a composite image 251 according to the embodiment. Cardboard box image O1 is a fallen object image whose position and size were specified by the worker and received by the reception unit 23. Cardboard box image O2 is a fallen object image whose position was specified by point P11 and whose size was determined by the determination unit 24. The position of cardboard box image O2 is determined, for example, so that its lower left corner overlaps the position of point P11.

[0045] Furthermore, although the shooting conditions such as the time of shooting, weather conditions at the time of shooting, and season of shooting differ for each of the multiple background images 11 of the road 500 captured by camera 1, the field of view and focal length are the same. Therefore, the position and size determined by the determination unit 24 can be applied to other background images 11 as well. Thus, when the generation unit 25 determines the position and size of a fallen object for a certain background image 11, it can generate a composite image in which the same position and size of fallen object is placed for other background images 11. The composite image 251 generated in this way by the generation unit 25 is used as training data for the learning model 221.

[0046] <Processing Flow> Figure 12 is a diagram showing an example of the processing flow of the detection device 2 according to the embodiment. Figure 12 illustrates the processing flow of the process for generating training data for the learning model 221. The following describes an example of the processing flow of the detection device 2 with reference to Figure 12. The processing flow by the detection device 2 illustrated in Figure 12 is an example of an "image processing method". The program that causes the detection device 2 to execute the processing flow illustrated in Figure 12 is an example of an "image processing program".

[0047] In step S1, the reception unit 23 accepts the specification of the type, location, and size of the falling object. If multiple falling objects are specified, the reception unit 23 accepts the specification of the location and size of one of the falling objects (referred to as the first falling object). Then, the reception unit 23 accepts the specification of the location to be placed for the other falling object (referred to as the second falling object), which is different from the first falling object. In other words, the reception unit 23 does not need to accept the specification of the size of the second falling object.

[0048] In step S2, the determination unit 24 obtains the distance indicated by the width of one pixel at the location where the first falling object is placed, based on the position of the first falling object received in step S1 and the distance map 263. The determination unit 24 determines the width (length) of the first falling object based on the number of pixels of the width of the first falling object and the distance indicated by the width of one pixel. The determination unit 24 also calculates the ratio of the height to the width of the first falling object based on the number of pixels of the width of the first falling object and the number of pixels of the height. The determination unit 24 obtains the distance indicated by the width of one pixel at the location of the second falling object from the distance map 263, and determines the number of pixels of the width of the second falling object based on the obtained distance indicated by the width of one pixel and the length of the width of the first falling object. Then, the determination unit 24 determines the number of pixels of the height of the second falling object based on the ratio of the height to the width of the first falling object and the number of pixels of the width of the second falling object.

[0049] In step S3, a composite image 251 is generated by superimposing the first and second falling objects, which are located at the positions and sizes determined in step S2.

[0050] <Effects of the Embodiment> According to this embodiment, if the position and size of one falling object are specified, when placing falling objects in other locations, only the position needs to be specified, and the size does not need to be specified. Therefore, compared to specifying the position and size of each falling object, the amount of work involved in generating training data is reduced.

[0051] In this embodiment, the reception unit 23 receives the location of other falling objects as shown at point P11 in Figure 10. Therefore, according to this embodiment, a composite image 251 can be generated in which the image of the falling object is placed at the worker's desired location.

[0052] Furthermore, according to this embodiment, once the position and size of the falling object to be superimposed on the background image 11 captured by camera 1 are determined by the reception unit 23 and the determination unit 24, a composite image can be generated in which the falling object of the position and size determined by the reception unit 23 and the determination unit 24 is also placed on other background images 11 captured by camera 1. Therefore, according to this embodiment, multiple composite images 251 can be generated with the workload required to generate one composite image 251. Note that, for example, if camera 1 is a pan-tilt-zoom (PTZ) camera, the background images 11 captured by camera 1 may include images with different angles of view, focal lengths, and shooting directions of camera 1. In such cases, for example, the angle of view, focal length, and shooting direction of camera 1 can also be associated with the background image ID in the background image management table 261. Then, the generation unit 25 selects a background image 11 with the same field of view, focal length, and shooting direction as the background image 11 whose position and size of the falling object have been determined by the reception unit 23 and the determination unit 24, based on the background image management table 261, and superimposes the falling object onto the selected background image 11.

[0053] <Variation> In the embodiments described above, the location of each falling object was specified by the worker, but it is not necessary to specify the location of all falling objects. For example, when the location and size of the first falling object are specified, the reception unit 23 may determine a random location on the road 500 as the location of the other falling objects. The determination unit 24 can then determine the size of the falling objects at the locations randomly determined by the reception unit 23 using the distance map 263. By performing this process, it is no longer necessary for the worker to specify the location of all falling objects, thus further reducing the workload in generating training data.

[0054] However, if the reception unit 23 randomly determines the location of the fallen object, there is a risk that the object may be placed outside the road 500. Therefore, the reception unit 23 should identify the area of ​​the road 500 using image recognition and determine the location of the fallen object within the identified area.

[0055] In the embodiments described above, the falling object detection system 100 was described as an example, but the systems to which the disclosed technology is applied are not limited to the falling object detection system 100. The disclosed technology can be applied to various systems that detect such objects by image processing, for example. Examples of systems to which the disclosed technology can be applied include intruder detection systems and robot vision systems.

[0056] The embodiments and variations disclosed above can be combined in any way.

[0057] <Computer-readable recording medium> An information processing program that enables a computer or other machine or device (hereinafter referred to as "computer, etc.") to perform any of the above functions can be recorded on a recording medium that the computer, etc. can read. By having the computer, etc. read and execute the program on this recording medium, it can be made to provide that function.

[0058] Here, a recording medium that can be read by a computer, etc., refers to a recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical means and can be read by a computer, etc. Among such recording media, a computer, etc. Removable media include, for example, flexible disks, magneto-optical disks, Compact Disc Read Only Memory (CD-ROM), Compact Disc-Recordable (CD-R), Compact Disc-ReWriterable (CD-RW), Digital Versatile Disc (DVD), Blu-ray Disc (BD), Digital Audio Tape (DAT), 8mm tape, flash memory, external hard disk drives, and Solid State Drives (SSDs). Additionally, there are internal hard disk drives, SSDs, and ROMs, which are recording media fixed to computers and other devices.

[0059] <Note 1> A storage unit (203) that stores a captured image (11) of the target area (500), and a correspondence relationship (263) between each position on the captured image (11) and the distance on the target area indicated by the unit pixel at each position, It comprises a control unit (201) and, The control unit (201) is, A reception process that accepts the specification of a first size at a first position of an object image (12, O1) to be placed on the captured image (11), A determination process to determine the second size of the object image (12, O2) at a second position (P11) different from the first position on the captured image (11) based on the first size and the correspondence relationship (263) stored in the storage unit (263), The process involves generating a composite image (251) in which an object image (12, O1) of a first size is placed at the first position on the captured image (11), and an object image (12, O2) of a second size is placed at the second position on the captured image (11). Image processing device (2). <Note 2> The aforementioned reception process includes a process for receiving the designation of the second position, The image processing apparatus according to claim 1. <Note 3> The reception process includes a process for determining the second position to be within a predetermined area in the captured image. The image processing device (2) described in Appendix 1. <Note 4> The memory unit (203) stores a plurality of the captured images having the same angle of view, focal length, and shooting direction. The aforementioned generation process is A process to generate a first composite image (251) in which the object image of the first size is placed at the first position on the first captured image (11) and the object image of the second size is placed at the second position on the first captured image, The process includes: placing the object image of a first size at a first position on a second captured image, which is different from the first captured image among the captured images (11), and generating a second composite image (251) in which the object image of a second size is placed at a second position on the second captured image, Image processing device (2) as described in any one of the appendices 1 to 3. <Note 5> A computer (2) has a storage unit (203) that stores a captured image (11) of a target area (500), and a correspondence (263) between each position on the captured image (11) and the distance on the target area indicated by the unit pixel at each position, A reception process that accepts the specification of a first size at a first position of an object image (12, O1) to be placed on the captured image (11), A determination process to determine the second size of the object image (12, O2) at a second position (P11) different from the first position on the captured image (11) based on the first size and the correspondence relationship (263) stored in the storage unit (263), The process involves generating a composite image (251) in which an object image (12, O1) of a first size is placed at the first position on the captured image (11), and an object image (12, O2) of a second size is placed at the second position on the captured image (11). Image processing methods. <Note 6> A computer (2) has a storage unit (203) that stores a captured image (11) of a target area (500), and a correspondence (263) between each position on the captured image (11) and the distance on the target area indicated by the unit pixel at each position, A reception process that accepts the specification of a first size at a first position of an object image (12, O1) to be placed on the captured image (11), A determination process to determine the second size of the object image (12, O2) at a second position (P11) different from the first position on the captured image (11) based on the first size and the correspondence relationship (263) stored in the storage unit (263), The process involves generating a composite image (251) by placing the object image (12, O1) of the first size at the first position on the captured image (11) and placing the object image (12, O2) of the second size at the second position on the captured image (11). Image processing program. [Explanation of symbols]

[0060] 1. Camera 2. Detection device 3...Display device 11. Background image 12. Images of falling objects 21··Acquisition part 22...Extraction part 23. Reception Department 24. Decision Section 25...Generation part 26. Management Database 100. Falling Object Detection System 201··CPU 202...Main memory 203...Auxiliary storage section 204 Communications Department 205...Connection terminals 221 ··Learning Model 231...Falling object designation screen 251 ··Composite image 261. Background Image Management Table 262 ·· Falling Object Management Table 263 Distance Map 500...road 2311 ··Image display area 2312 ··Falling Object Selection Menu 2313 ·· Rectangular frame B1...Bus K1...Traffic warden L1 Connection Cable M1... Background Model N1 Computer Network O1...Cardboard box image O2 cardboard box image

Claims

1. A storage unit that stores a captured image of the target area, the correspondence between each position on the captured image and the distance on the target area indicated by the unit pixel at each position, It comprises a control unit and, The control unit, A reception process that accepts the specification of a first size at a first position of an object image to be placed on the captured image, A determination process for determining the second size of the object image at a second position different from the first position on the captured image, based on the first size and the correspondence relationship stored in the storage unit, The process involves generating a composite image in which the object image of the first size is placed at the first position on the captured image, and the object image of the second size is placed at the second position on the captured image. Image processing device.

2. The aforementioned reception process includes a process for receiving the designation of the second position, The image processing apparatus according to claim 1.

3. The reception process includes a process for determining the second position to be within a predetermined area in the captured image. The image processing apparatus according to claim 1.

4. The memory unit stores a plurality of the captured images having the same angle of view, focal length, and shooting direction. The aforementioned generation process is A process to generate a first composite image in which the object image of the first size is placed at the first position on the first captured image, and the object image of the second size is placed at the second position on the first captured image, The process includes: placing the object image of the first size at the first position on a second captured image, which is different from the first captured image among the captured images, and generating a second composite image in which the object image of the second size is placed at the second position on the second captured image. The image processing apparatus according to any one of claims 1 to 3.

5. A computer having a storage unit that stores a captured image of a target area, and the correspondence between each position on the captured image and the distance on the target area indicated by the unit pixel at each position, A reception process that accepts the specification of a first size at a first position of an object image to be placed on the captured image, A determination process for determining the second size of the object image at a second position different from the first position on the captured image, based on the first size and the correspondence relationship stored in the storage unit, The process involves generating a composite image in which the object image of the first size is placed at the first position on the captured image, and the object image of the second size is placed at the second position on the captured image. Image processing methods.

6. A storage unit that stores a captured image of a target area, and the correspondence between each position on the captured image and the distance on the target area indicated by the unit pixel at each position. On the computer, A reception process that accepts the specification of a first size at a first position of an object image to be placed on the captured image, A determination process for determining the second size of the object image at a second position different from the first position on the captured image, based on the first size and the correspondence relationship stored in the storage unit, The process involves generating a composite image in which the object image of the first size is placed at the first position on the captured image, and the object image of the second size is placed at the second position on the captured image. Image processing program.

Images