Information processing device, information processing system, control method for information processing device, and program

The information processing device optimizes label placement within polygonal areas on maps to prevent object obscuration and overlap, enhancing visibility and attribution by determining optimal display positions and adjusting label size or orientation.

JP2024157845AInactive Publication Date: 2024-11-08CANON KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023072455
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-26
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When objects of interest are located within polygonal areas on a map, displaying labels inside the polygons can obscure the visibility of these objects, and overlapping labels from adjacent polygons further reduce visibility.

Method used

An information processing device that acquires the likelihood of object occlusion by labels, determines optimal label display positions based on occlusion likelihood and distance from other labels, and adjusts label size or orientation to minimize overlap and obstruction.

Benefits of technology

Prevents the visibility of detection targets from being decreased by label display, ensuring clear attribution and reducing overlap with adjacent labels.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024157845000001_ABST
    Figure 2024157845000001_ABST
Patent Text Reader

Abstract

To make it possible to prevent a decrease in visibility of a detection target in an image due to label display.SOLUTION: An information processing device includes: first acquisition means for acquiring a likelihood that a detection target exists for each region of an image divided into a plurality of regions; second acquisition means for acquiring an occlusion likelihood representing a likelihood of the detection target being occluded by a label for each region of the image based on the size of the label that displays information related to a detection region set for the image and the likelihood acquired by the first acquisition means; determination means for determining a display position of the label in the image based on the occlusion likelihood acquired by the second acquisition means; and display means for displaying the label corresponding to the detection region at the display position of the label determined by the determination means.SELECTED DRAWING: Figure 7
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing system, a control method for an information processing device, and a program. [Background technology]

[0002] Conventionally, in cases where the boundary of an area is a polygon, such as in a map, information such as the land lot number and building name of the land has been displayed as a label inside the polygon to ensure the attribution and visibility of the polygon and the label. This is because the focus of attention on a map is the edge of the polygon that is the boundary of the area, and there is no problem even if the inside of the polygon is hidden by a label, etc. Patent Document 1 discloses a technology related to a method for reliably and quickly calculating the placement position of a label inside a polygonal figure. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7015879 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the purpose of placing a label in a polygonal region may be, as in object detection, when an object is present within the polygonal region and information related to the result is displayed as a label. In this case, since the target of interest is within the polygonal detection region, there is a problem that if a label is displayed inside the polygon, the target of interest within the detection region is occluded by the label, and the visibility of the target of interest within the detection region is reduced. When displaying the detection result as a label in a polygonal detection region, placing the label outside the polygon prevents the target of interest within the detection region from being occluded. However, if there is an adjacent polygonal detection region, there is a risk that the target of interest and other labels will overlap in the detection region of the adjacent polygon, resulting in reduced visibility. The present invention has been made in view of the above circumstances, and has an object to prevent a decrease in visibility of detection targets in an image due to label display. [Means for solving the problem]

[0005] The information processing device of the present invention is characterized in having a first acquisition means for acquiring a likelihood that a detection target exists in each of a plurality of regions of an image divided into the plurality of regions, a second acquisition means for acquiring an occlusion likelihood representing the likelihood of the detection target being occluded by the label for each of the regions of the image based on the size of a label displaying information related to a detection region set for the image and the likelihood acquired by the first acquisition means, a determination means for determining a display position of the label in the image based on the occlusion likelihood acquired by the second acquisition means, and a display means for displaying the label corresponding to the detection region at the display position of the label determined by the determination means. Effect of the Invention

[0006] According to the present invention, it is possible to prevent a decrease in visibility of a detection target in an image caused by label display. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an example of a configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an imaging apparatus. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging apparatus. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 5] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device. [Figure 6] 1A and 1B are diagrams illustrating an example of a captured image and a detection area. [Figure 7] 11 is a flowchart illustrating an example of an image analysis process. [Figure 8]13 is a flowchart illustrating an example of a process for determining a label display position. [Figure 9] 13A and 13B are diagrams showing an example of a screen display after the label display positions in four detection regions have been determined, and an example of a label for a fifth detection region. [Figure 10] 13A to 13C are diagrams illustrating an example of a cropped image, a likelihood map, an occlusion likelihood map, and a display position score. [Figure 11] FIG. 13 is a diagram showing an example of a screen display in which the label display positions of all labels have been decided. [Figure 12] 13 is a flowchart illustrating an example of a process for determining a label display position. [Figure 13] 1A and 1B are diagrams illustrating an example of a cropped image, a likelihood map, and an occlusion likelihood map. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the embodiment described below does not limit the invention according to the claims. Although multiple features are described in the embodiment, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the attached drawings, the same reference numerals are used for the same or similar configurations, and duplicated explanations are omitted.

[0009] <System configuration> FIG. 1 is a diagram showing an example of the configuration of an information processing system 101 in this embodiment. In the following, an example will be described in which the information processing system in this embodiment is applied to a parking situation determination system. However, the present invention is not limited to this, and the information processing system in this embodiment can be applied to any system that analyzes images and outputs predetermined information. The information processing system 101 includes imaging devices 110a to 110d, a network 120, and an information processing device 130. In the following, the imaging devices 110a to 110d will be referred to as "imaging devices 110".

[0010] The imaging device 110 is an imaging device such as a network camera that captures an image. In this embodiment, the imaging device 110 is assumed to have a built-in arithmetic device capable of processing video, but is not limited thereto. For example, an external computer such as a PC (personal computer) connected to the imaging device 110 may exist, and a combination of these may be treated as the imaging device 110. Note that, in the example shown in FIG. 1, four imaging devices 110 are shown, but this is not limiting, and the number of imaging devices 110 in the information processing system 101 is arbitrary. The information processing device 130 is a computer such as a PC, and has an image analysis processing function. In addition, the information processing device 130 accepts input from a user, outputs information to the user (for example, displays information), and the like.

[0011] The imaging device 110 and the information processing device 130 are connected to each other via a network 120 so as to be able to communicate with each other. The network 120 includes a plurality of routers, switches, cables, and the like that meet a communication standard such as Ethernet (registered trademark). In this embodiment, the network 120 may be any network that enables communication between the imaging device 110 and the information processing device 130, and may be constructed with any scale, configuration, and conforming communication standard. For example, the network 120 may be the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), or the like. In addition, the network 120 may be configured so as to enable communication with a communication protocol that complies with the ONVIF (Open Network Video Interface Forum) standard. However, the present invention is not limited to this, and the network 120 may be configured so as to enable communication with another communication protocol such as a unique communication protocol.

[0012] <Configuration of imaging device> The following describes the configuration of the imaging device 110. Fig. 2 is a diagram showing an example of the hardware configuration of the imaging device 110. The imaging device 110 has, as its hardware configuration, an imaging unit 201, an image processing unit 202, an arithmetic processing unit 203, and a distribution unit 204, for example.

[0013] The imaging unit 201 has a lens unit for forming an image of light, and an imaging element for converting the formed light into an analog signal. The lens unit has a zoom function for adjusting the angle of view, an aperture function for adjusting the amount of light, and the like. The imaging element has a gain function for adjusting the sensitivity when converting light into an analog signal. These functions are adjusted based on setting values ​​notified by the image processing unit 202. The analog signal acquired by the imaging unit 201 is converted into a digital signal by an analog-digital conversion circuit, and transferred to the image processing unit 202 as an image signal.

[0014] The image processing unit 202 has an image processing engine and its peripheral devices. The peripheral devices include, for example, a RAM (Random Access Memory) and drivers for each interface. The image processing unit 202 performs image processing, such as development processing, filter processing, sensor correction, and noise removal, on the image signal acquired from the imaging unit 201 to generate image data. The image processing unit 202 can also transmit setting values ​​to the lens unit and image sensor of the imaging unit 201 and perform exposure adjustment so that an appropriately exposed image can be acquired. The image data generated in the image processing unit 202 is transferred to the arithmetic processing unit 203.

[0015] The arithmetic processing unit 203 is composed of one or more processors such as a CPU or MPU, memories such as a RAM or ROM, and drivers for each interface, and executes, for example, predetermined processing to be performed in the imaging device 110. Note that CPU is an acronym for Central Processing Unit, MPU is an acronym for Micro Processing Unit, RAM is an acronym for Random Access Memory, and ROM is an acronym for Read Only Memory. Image data received from the image processing unit 202 and data resulting from processing by the arithmetic processing unit 203 are transferred to the distribution unit 204.

[0016] The distribution unit 204 includes a network distribution engine and peripheral devices such as a RAM and an ETH PHY module. The ETH PHY module is a module that executes processing of the physical (PHY) layer of Ethernet. The distribution unit 204 converts the image data and processing result data acquired from the arithmetic processing unit 203 into a format that can be distributed to the network 120, and outputs the converted data to the network 120.

[0017] 3 is a diagram showing an example of the functional configuration of the imaging device 110. The imaging device 110 has, as its functions, an imaging control unit 301, a signal processing unit 302, a storage unit 303, a control unit 304, an analysis unit 305, and a network communication unit 306, for example.

[0018] The imaging control unit 301 executes control to capture the surrounding environment via the imaging unit 201. The signal processing unit 302 performs a predetermined process on the image captured by the imaging control unit 301 to generate data of the captured image. Hereinafter, the data of the captured image is simply referred to as a "captured image". For example, the signal processing unit 302 encodes the image captured by the imaging control unit 301. For a still image, the signal processing unit 302 encodes the image using an encoding method such as JPEG (Joint Photographic Experts Group). For a moving image, the signal processing unit 302 encodes the image using an encoding method such as H.264 / MPEG-4 AVC or HEVC (High Efficiency Video Coding). For example, the signal processing unit 302 may encode the image using an encoding method selected by a user from a plurality of encoding methods set in advance via an operation unit (not shown) of the imaging device 110.

[0019] The storage unit 303 stores data related to various processes executed by the imaging device 110. The control unit 304 controls the signal processing unit 302, the storage unit 303, the analysis unit 305, and the network communication unit 306 so that each performs a predetermined process. The analysis unit 305 performs image analysis processing on the captured image. The network communication unit 306 communicates with the information processing device 130 via the network 120.

[0020] <Configuration of information processing device> The configuration of the information processing device 130 will be described. Fig. 4 is a diagram showing an example of the hardware configuration of the information processing device 130. The information processing device 130 has, for example, a processor 401, a RAM 402, a ROM 403, a storage unit 404, an input unit 405, a display unit 406, a communication interface (communication I / F) 407, and a system bus 408 as its hardware configuration. The processor 401, the RAM 402, the ROM 403, the storage unit 404, the input unit 405, the display unit 406, and the communication I / F 407 are communicatively connected via the system bus 408. Note that the information processing device 130 may further have other configurations.

[0021] The processor 401 is, for example, a CPU, and is responsible for overall control of the information processing device 130. The processor 401 controls the operation of each unit connected via, for example, a system bus 408. The processor 401 may be an MPU or a GPU (Graphics Processing Unit). The RAM 402 has functions as the main memory, work area, etc. of the processor 401, and is a memory that temporarily stores programs, data, etc. that are supplied. The ROM 403 is a memory that stores programs, data, etc. that the processor 401 uses for processing.

[0022] The storage unit 404 stores, for example, a program related to the processing performed by the processor 401, various data required for performing the processing related to the program, etc. The storage unit 404 is, for example, a storage device such as an HDD (hard disk drive) or an SSD (solid state drive). The input unit 405 has operation members such as a mouse, a keyboard, and a button, and inputs user operations to the information processing device 130. The display unit 406 has a display member such as a liquid crystal display, and displays the results of processing by the processor 401, etc. The communication I / F 407 connects the information processing device 130 to the network 120, and controls communication with other devices such as the imaging device 110.

[0023] The information processing device 130 can execute various functions of the information processing device 130 shown in FIG. 5 and processes of flowcharts described later by the processor 401 executing processes based on programs stored in the ROM 403, the storage unit 404, etc.

[0024] 5 is a diagram showing an example of a functional configuration of the information processing device 130. The information processing device 130 has, for example, a network communication unit 501, a control unit 502, a display unit 503, an operation unit 504, an analysis unit 505, and a storage unit 506 as its functional configuration. The network communication unit 501 is connected to, for example, a network 120, and executes communication with an external device such as the imaging device 110 via the network 120. Note that this is merely an example, and for example, the network communication unit 501 may be configured to establish a direct connection with the imaging device 110 and communicate with the imaging device 110 without going through the network 120 or other devices. The control unit 502 controls the network communication unit 501, the display unit 503, the operation unit 504, the analysis unit 505, and the storage unit 506 to execute their respective processes.

[0025] The display unit 503 presents information to the user, for example, via a display. In this embodiment, for example, the information is presented to the user by displaying the result of rendering by the browser on the display. Note that the information may be presented by a method other than a screen display, such as sound or vibration. The operation unit 504 accepts operations from the user. In this embodiment, as an example, the operation unit 504 is a mouse or a keyboard, and the user operates these to input user operations to the browser. However, this is not limited to this, and the operation unit 504 may be any device that can detect the intention of other users, such as a touch panel or a microphone.

[0026] The analysis unit 505 executes image analysis processing. The analysis unit 505 is an example of a first acquisition means, a second acquisition means, a third acquisition means, and a determination means. In this embodiment, the analysis unit 505 executes, for example, vehicle detection processing and processing related to label display on an input image as image analysis processing. In this embodiment, since the applied system is a parking situation determination system, the analysis unit 505 executes vehicle detection processing, but is not limited to this. The analysis unit 505 may execute object detection processing other than a vehicle as image analysis processing depending on the applied system, etc., and may also execute determination processing of a specific object.

[0027] In this embodiment, as an example, a vehicle detector used in the vehicle detection process uses an object detection model trained by a technique called machine learning, which finds patterns hidden in a large amount of data through repeated calculations. For example, the technology described in the following document A is applied as the object detection model.

[0028] (Reference A) RRSelvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, “Grad-CAM:Visual Explanations From Deep Networks via Gradient-Based Localization”, Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017

[0029] An image is divided into a number of regions, and the probability density that a vehicle exists in each region is evaluated using observed values ​​and called the likelihood. A map in which the likelihood of each region is mapped to the entire image in a range of 0 to 1 is called a likelihood map. In vehicle detection processing, the analysis unit 505 takes an image as input and outputs a likelihood map. In the likelihood map output by vehicle detection processing, a region determined to have a high probability of a vehicle existing shows a high likelihood (approaching 1), and a region determined to have a high probability of no vehicle existing shows a low likelihood (approaching 0).

[0030] The storage unit 506 stores a vehicle detector (object detection model related to a vehicle), a detection area described below, label information, and the like. Here, a label is a rectangular area that displays information such as a detection result, time, and quantity on the screen. Note that the shape of the label area is not limited to a rectangular area, and can be an area of ​​any shape that can display information. The label information includes information regarding a character string to be displayed, the number of characters, the character size, and the margin size, and the like.

[0031] <Processing flow> Next, an example of the flow of processing executed by the information processing system 101 will be described. Each of the following processes is realized by the processor 401 of the information processing device 130 executing a program stored in a memory or the like. However, this is merely an example, and some or all of the processes described below may be realized by the imaging device 110 or dedicated hardware.

[0032] (Image analysis processing) The image analysis process executed by the information processing device 130 will be described below. FIG. 6(a) is a diagram for explaining an example of a captured image 601 acquired from the imaging device 110, and FIG. 6(b) is a diagram for explaining an example of six detection areas 602 to 607 set as polygons. The captured image 601 shown in FIG. 6(a) is an example of an image obtained by capturing a part of a parking lot. The detection areas 602 to 607 shown in FIG. 6(b) are polygonal areas set corresponding to each parking area, and in this embodiment, the detection areas are labeled with information related to the corresponding detection areas. Each polygonal area set as a detection area is expressed by a set of vertex coordinates, and for example, the polygonal area is defined by connecting the points indicated by the vertex coordinates with sides in the order of the vertex coordinates. The coordinate information of the detection areas 602 to 607 is stored in the storage unit 506, and can be edited via a setting screen or the like. For example, a setting screen related to the coordinate information of the detection area may be displayed on the display unit 406 of the information processing device 130, and the coordinate information of the detection area may be saved by the user setting the coordinate information using the input unit 405. Also, for example, if the information processing device 130 is a server, the coordinate information of the detection area may be saved by setting the detection area from a client (not shown) such as a PC using a setting function provided by the server. Note that, although an example in which the number of detection areas is six is ​​shown in Fig. 6(b), the number of detection areas that can be set is not limited to six, and any number of detection areas can be set.

[0033] Fig. 7 is a flowchart illustrating an example of image analysis processing executed by the information processing device 130. The processing of the flowchart shown in Fig. 7 is repeatedly executed on an input image (captured image) captured by the imaging device 110 and input to the information processing device 130.

[0034] In step S701, the analysis unit 505 of the information processing device 130 acquires the captured image 601 from the imaging device 110. In step S702, the analysis unit 505 executes vehicle detection processing on the captured image 601 acquired in step S701 using a vehicle detector, and acquires a likelihood map indicating the likelihood of the presence of a vehicle to be detected for each region of the captured image 601 divided into a plurality of regions. Here, the vehicle detection processing may be performed by cropping the captured image into small pieces so as to include the detection region (polygonal region) for each detection region, and executing detection processing on each cropped image to acquire a likelihood map for each cropped image. Alternatively, the vehicle detection processing may be performed by processing the captured image as it is, i.e., executing detection processing on the entire captured image to acquire one likelihood map for the entire captured image.

[0035] In step S703, the analysis unit 505 acquires the label size of each label based on information such as character information (character string and number of characters) to be displayed on the label, character size, and margin size. The acquired label size is stored in the storage unit 506 as label information. For example, when the parking time is displayed as "1:00", if the number of characters is 4, the character size per character is 15 pixels, and the margin sizes on the top, bottom, left, and right are 5 pixels, the label size of this label is 70 pixels wide and 25 pixels high. In addition, the information to be displayed on the label may be likelihood, time, parking lot number, vehicle type, license plate, number of parked cars, parking lot vacancy information, etc. In addition, if the result of the detection process in step S702 is not reflected in the displayed character information and the character information to be displayed on the label does not change, the label size does not change depending on the input image, so the process of this step is performed only the first time and does not necessarily need to be performed thereafter.

[0036] In step S704, the analysis unit 505 executes a process of determining a label display position, and determines a label display position for each detection region based on the likelihood map acquired in step S702 and information related to the detection regions 602 to 607 stored in the storage unit 506. The process of determining the label display position in step S704 will be described in detail later.

[0037] In step S705, the display unit 503 displays a label for each detection area based on the label display position determined in step S704, and superimposes the label on the captured image 601. Here, the label may be displayed by being directly superimposed on the video stream delivered from the imaging device 110, or may be displayed by being superimposed on the video stream on a UI screen using a browser or the like.

[0038] (Decision process for label display position) Fig. 8 is a flowchart for explaining an example of the process of determining the label display positions in step S704 in Fig. 7. In the following, an example will be described in which the display positions of the labels 901-904 belonging to four detection areas 602-605 of the six detection areas 602-607 as shown in Fig. 9(a) are determined, and then the display position of the label 905 shown in Fig. 9(b) belonging to the detection area 606 is determined. However, the order in which the display positions of the labels are determined is not limited to this, and for example, the display positions may be determined in order starting from the label belonging to the detection area with the smallest area.

[0039] The process of determining the label display position will be described with reference to Fig. 10(a) to Fig. 10(g) as appropriate. Fig. 10(a) shows an example of a cropped image 1001 obtained by dividing a captured image 601 into 14 horizontally and 11 vertically into 14 x 11 regions, and cropping the image into a small piece that includes a detection region 606. Hereinafter, each divided region is called a cell, and is represented by a combination of columns A to N and rows 1 to 11. Fig. 10(b) shows an example of a likelihood map 1002, and Fig. 10(c), Fig. 10(e), and Fig. 10(g) show examples of occlusion likelihood maps 1003, 1005, and 1007, respectively, calculated in the process of determining the label display position. Fig. 10(d) shows an example of a display position score (polygon display position score) 1004, and Fig. 10(f) shows an example of a display position score (overall display position score) 1006.

[0040] In step S801, the analysis unit 505 generates an occlusion likelihood map in which an occlusion likelihood that indicates the likelihood of a detection target being occluded by a label is mapped based on the label information stored in the storage unit 506 and the likelihood map acquired in step S702. The occlusion likelihood is, for example, a weighted average value according to the overlapping area of ​​the likelihood at the position where the label and the likelihood map overlap. The occlusion likelihood map 1003 illustrated in FIG. 10(c) is generated based on a likelihood map 1002 that is a detection result obtained by performing vehicle detection processing on a cropped image 1001. Here, the occlusion likelihood Lxy when a label is placed at a position whose center coincides with the center of cell XY is calculated by the following formula, where L0xy is the likelihood in each cell and R0xy is the area ratio occluded by the label (the sum of the area ratios occluded by the label is 1). It indicates that the higher the occlusion likelihood, the higher the possibility that the detection target will be occluded by the label when the label is displayed at that position.

[0041]

number

[0042] The value of cell E6 in the occlusion likelihood map 1003 is the average likelihood of occlusion when the label 905 is arranged so that the center of the label coincides with the center of cell E6 in the likelihood map 1002. For example, the area ratio of each cell occluded by the label 905 when the label 905 is arranged so that the center of the label 905 coincides with the center of cell E6 is 0.5 for cell E6, 0.25 for cells D6 and F6, and 0 for other cells. Here, the maximum likelihood value of 1 is substituted for the region outside the cropped image where the likelihood is not calculated (cell D6 in the above example) to calculate the occlusion likelihood, thereby preventing the label from being placed at a position away from the detection region in the subsequent processing. By processing as described above, the occlusion likelihood of cell E6 in the occlusion likelihood map 1003 is calculated to be 0.53. Although the occlusion likelihood at each position is exemplified as 0.0 to 1.0, the number of divisions, the upper and lower limits of occlusion likelihood, and the number of stages are not limited to this. For example, the number of divisions may be the same as or less than the resolution of the captured image, and the occlusion likelihood may have two stages, 0 and 1.

[0043] In step S802, the analysis unit 505 adds, for each cell, a display position score (polygon display position score) 1004 indicating the priority of the label display position in the detection area of ​​the polygon to the occlusion likelihood map 1003 generated in step S801. The polygon display position score 1004 illustrated in FIG. 10(d) is lowered toward the lower part of the area overlapping with the cropped image 1001 and is higher toward the upper part, thereby increasing the display priority of the label in the lower part of the detection area. However, this is not limited to this, and the priority of the label display position may be determined by the user. Also, a different polygon display position score may be used for each polygon, and further, for example, the priority of the polygon display position score of a specific position may be increased. Also, it is possible to arrange the label so that it is included inside the polygon by lowering the score of the position where the label is included in the polygon and increasing the score of the position outside the polygon. FIG. 10E shows an occlusion likelihood map 1005 calculated by adding up the occlusion likelihood map 1003 and the polygon display position score 1004 for each cell.

[0044] In step S803, the analysis unit 505 sums the display position score (overall display position score) 1006, which indicates the priority of the label display position according to the distance from other polygon positions and other label positions, to the occlusion likelihood map 1005 calculated in step S802 for each cell. The overall display position score 1006 illustrated in FIG. 10(f) is obtained by dividing the captured image into 14×11, and the score is increased as the overlap with polygons or other labels related to other detection areas increases, and the score is decreased in other areas. This prevents the label from overlapping with other polygons or other labels. For example, by using an overall display position score that increases the priority of the label display position as the distance from other labels whose display positions have been determined increases, it is possible to prevent the label from overlapping with other labels. In addition, the number of divisions and the number of stages of the overall display position score are not limited to these. For example, the number of divisions may be the same as or less than the resolution of the captured image, and the overall display position score may be two stages, 0 and 1. FIG. 10( g ) shows an occlusion likelihood map 1007 calculated by adding up the occlusion likelihood map 1005 and the overall display position score 1006 for each cell.

[0045] In step S804, the analysis unit 505 determines whether the minimum value of the occlusion likelihood in the occlusion likelihood map 1007 calculated in step S803 is below a predetermined value. If the analysis unit 505 determines that the minimum value of the occlusion likelihood in the occlusion likelihood map 1007 is not below the predetermined value (NO in step S804), the process proceeds to step S805. On the other hand, if the analysis unit 505 determines that the minimum value of the occlusion likelihood in the occlusion likelihood map 1007 is below the predetermined value (YES in step S804), the process proceeds to step S806.

[0046] In step S805, the analysis unit 505 reduces the size of the label or rotates the orientation of the label, and stores the result as label information in the storage unit 506. After performing the process in step S805, the process returns to step S801. Here, reducing the size of the label indicates reducing at least one of the character size and the margin size, and the reduction rate in one process is, for example, between 95% and 50%. The analysis unit 505 performs both or either of the size reduction and orientation rotation of the label. Furthermore, when the process in step 805 is performed multiple times for the same label, the analysis unit 505 may change whether to perform both or either of the size reduction and orientation rotation depending on the number of times the process is performed.

[0047] If the minimum value of the occlusion likelihood does not fall below a predetermined value in step S804 even after the process of step S805 is executed a predetermined number of times or if the label size is equal to or smaller than a predetermined size, the label display position may be determined to be a predetermined, specified label display position. This specified label display position may be stored in the storage unit 506. If the label display position is determined to be the specified label display position, the analysis unit 505 may omit execution of the process of step S806 and execute the process of step S807.

[0048] In step S806, the analysis unit 505 determines the label display position based on the occlusion likelihood map 1007 calculated in step S803. The analysis unit 505 determines a position where the detection region and the label at least partially overlap and the occlusion likelihood is equal to or less than a predetermined value as the label display position. Furthermore, the analysis unit 505 may determine a position where the detection region and the label entirely overlap, that is, a position where the label is included in the detection region, as the label display position. Furthermore, the analysis unit 505 may determine a position where the occlusion likelihood is minimum from the occlusion likelihood map 1007 as the label display position, or may determine the label display position according to the priority order of the label display position from among positions where the occlusion likelihood is equal to or less than a predetermined value.

[0049] In step S807, the analysis unit 505 determines whether or not there is a label whose label display position has not been determined. If the analysis unit 505 determines that there is a label whose label display position has not been determined (YES in step S807), the process returns to step S801, and steps S801 to S807 are executed for the label whose label display position has not been determined. On the other hand, if the analysis unit 505 determines that there is no label whose label display position has not been determined (NO in step S807), the process of determining the label display position shown in FIG. 8 is terminated, and the process returns to the image analysis process shown in FIG. 7.

[0050] In the manner described above, the label display positions are determined for all labels, including the label 905 belonging to the detection area 606 and the label 1101 belonging to the detection area 607, and an example in which the labels 901 to 905 and 1101 are superimposed on the captured image 601 is shown in FIG.

[0051] According to this embodiment, a label for displaying information can be displayed near a polygon that is a detection area so as to reduce overlap with the detection target, and it is possible to prevent the visibility of the detection target from being reduced by label display. Also, a label belonging to the detection area can be displayed near the polygon that is a detection area, and it is possible to ensure the attribution of the detection area and the label.

[0052] <Variation 1> An example will be described in which the occlusion likelihood map 1007 calculated in step S803 of the flowchart shown in Fig. 8 is stored in the storage unit 506, and a statistical process is performed based on the history of the stored occlusion likelihood map to suppress fluctuations in the label display position. Fig. 12 is a flowchart illustrating another example of the process of determining the label display position in step S704 of Fig. 7.

[0053] In step S1201, the analysis unit 505 of the information processing device 130 generates an occlusion likelihood map based on the label information stored in the storage unit 506 and the likelihood map acquired in step S702. In step S1202, the analysis unit 505 adds the display position score (polygon display position score) 1004 to the occlusion likelihood map 1003 generated in step S1201. In step S1203, the analysis unit 505 adds the display position score (overall display position score) 1006 to the occlusion likelihood map 1005 calculated in step S1202.

[0054] In step S1204, the analysis unit 505 determines whether the minimum value of the occlusion likelihood in the occlusion likelihood map 1007 calculated in step S1203 is below a predetermined value. If the analysis unit 505 determines that the minimum value of the occlusion likelihood is not below the predetermined value (NO in step S1204), the process proceeds to step S1205. On the other hand, if the analysis unit 505 determines that the minimum value of the occlusion likelihood is below the predetermined value (YES in step S1204), the process proceeds to step S1206.

[0055] In step S1205, the analysis unit 505 reduces the size of the label or rotates the orientation of the label, and stores the result as label information in the storage unit 506. After performing the process in step S1205, the process returns to step S1201. The processes in steps S1201 to S1205 are similar to the processes in steps S801 to S805 in the flowchart shown in FIG. 8, respectively, and therefore a detailed description thereof will be omitted.

[0056] In step S1206, the analysis unit 505 stores in the storage unit 506 the occlusion likelihood map 1007 calculated in step S1203 and label information such as the size and orientation of the labels when the occlusion likelihood map 1007 was calculated.

[0057] In step S1207, the analysis unit 505 generates an average occlusion likelihood map based on the multiple occlusion likelihood maps 1007 stored in the storage unit 506. The analysis unit 505 acquires the occlusion likelihood maps 1007 for the number of statistical processing data from the storage unit 506, and generates the average occlusion likelihood map by averaging the occlusion likelihoods in the occlusion likelihood maps 1007 for the number of statistical processing data acquired for each cell.

[0058] In step S1208, the analysis unit 505 determines a label display position based on the average occlusion likelihood map generated in step S1207. Here, the analysis unit 505 may determine a position in the average occlusion likelihood map where the occlusion likelihood (average occlusion likelihood) is minimum as the label display position, or may determine the label display position according to the priority order of the label display position from among positions where the average occlusion likelihood is equal to or less than a predetermined value.

[0059] In step S1209, the analysis unit 505 determines whether or not there is a label whose label display position has not been determined. If the analysis unit 505 determines that there is a label whose label display position has not been determined (YES in step S1209), the process returns to step S1201, and executes the processes of steps S1201 to S1209 for the label whose label display position has not been determined. On the other hand, if the analysis unit 505 determines that there is no label whose label display position has not been determined (NO in step S1207), the label display position determination process shown in FIG. 12 is terminated, and the process returns to the image analysis process shown in FIG. 7.

[0060] This makes it possible to prevent the visibility of the detection target from decreasing due to label display while ensuring the attribution of the detection region and the label. Also, by determining the label display position using multiple occlusion likelihood maps based on the history of the occlusion likelihood maps stored in the storage unit 506, it becomes possible to display the label display position while suppressing the fluctuation of the label display position to a predetermined amount or less.

[0061] <Variation 2> In the above-described embodiment, a set of a detection area and a label belonging thereto may be displayed in any number of colors so that the detection area and the label belonging thereto can be distinguished from adjacent detection areas and the labels belonging thereto. In this way, the detection area and the label belonging thereto are displayed in the same color, and other adjacent detection areas and the labels belonging thereto are displayed in different colors, making it possible to clarify the attribution of the detection area and the label, and to easily distinguish the attribution of the detection area and the label.

[0062] For example, the analysis unit 505 of the information processing device 130 can be optimized by applying the technology described in the following document B, so that adjacent detection areas are in different groups, and thus multiple detection areas set in a captured image can be classified into five or fewer groups.

[0063] (Reference B) N. Robertson, DPSanders, P. Seymour, R. Thomas, “Efficiently four-coloring planar graphs”, Symposium on Theory of Computing (STOC), 1996

[0064] Then, in step S705 of the flowchart shown in Fig. 7, the display unit 503 displays the detection areas belonging to each group and the labels belonging thereto using the color corresponding to the group. For example, five colors, red, orange, yellow, green, and blue, are designated as the colors corresponding to each group, and the detection areas and the labels belonging thereto are displayed in different colors for each group. Note that the five colors are not limited to these and can be any colors, but it is preferable that the colors be easily distinguishable.

[0065] In this way, by applying the technology described in Literature B to classify multiple detection areas into five or fewer groups, it is possible to clarify the attribution of the detection areas and labels using up to five different colors, making it possible to easily identify the attribution of the detection areas and labels using fewer colors.

[0066] <Modification 3> In the above-described embodiment, a UI (user interface) unit capable of acquiring a user operation may be provided, and labels may be highlighted in accordance with the result of the UI operation, for example, as a process of mouse over. For example, when a label or detection area is selected or in a state equivalent thereto as a result of a user operation by the operation unit 504, the display unit 503 may highlight the label or the label belonging to the detection area by enlarging or visualizing the label. This makes it possible to improve the visibility of the selected label and reduce the concern that unselected labels may block other labels or detection targets.

[0067] <Variation 4> An example will be described in which the information processing device 130 further includes, as an object detection model, a license plate detector that detects the license plate of a vehicle in addition to a vehicle detector, and prevents a decrease in the visibility of the license plate.

[0068] In step S702 of the flowchart shown in Fig. 7, the analysis unit 505 of the information processing device 130 executes a detection process on the captured image to obtain a likelihood map of the license plate together with a likelihood map of the vehicle. Fig. 13(b) shows an example of a likelihood map 1301 of the license plate, which is a detection result obtained by performing a license plate detection process on the cropped image 1001 shown in Fig. 13(a).

[0069] Also, in step S801 of the flowchart shown in Fig. 8, the analysis unit 505 generates a license plate occlusion likelihood map 1302 shown in Fig. 13(c) based on the acquired license plate likelihood map 1301. Here, an example has been shown in which the vehicle and license plate occlusion likelihood maps are generated separately, but the vehicle likelihood map and the license plate likelihood map may be combined, and then the vehicle and license plate occlusion likelihood map may be generated based on the combined likelihood map.

[0070] Next, the analysis unit 505 adds the occlusion likelihood map 1302 of the license plate to the occlusion likelihood map 1003 of the vehicle, the polygon display position score 1004, and the whole display position score 1006 to calculate the occlusion likelihood map 1303. By executing the processes from step S804 onwards based on the occlusion likelihood map 1303, it is possible to prevent the license plate from being occluded by the label and improve the visibility of the license plate.

[0071] <Other embodiments> In the above-described embodiment, an example was described in which the image captured by the imaging device 110 was subjected to image analysis processing by the information processing device 130. However, the image captured by the imaging device 110 may be subjected to image analysis processing by the imaging device 110.

[0072] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0073] It should be noted that the above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0074] The disclosure of the present embodiment includes the following configurations, methods, etc. (Configuration 1) A first acquisition means for acquiring a likelihood that a detection target exists in each of a plurality of regions of an image divided into the plurality of regions; a second acquisition means for acquiring an occlusion likelihood representing a likelihood of the detection target being occluded by the label for each of the regions of the image, based on a size of a label indicating information related to a detection region set for the image and the likelihood acquired by the first acquisition means; a determination means for determining a display position of the label in the image based on the occlusion likelihood acquired by the second acquisition means; a display unit that displays the label corresponding to the detection area at a display position of the label determined by the determination unit. (Configuration 2) 2. The information processing device according to configuration 1, wherein the determining means determines a display position of the label at a position where the detection area and the label at least partially overlap and the occlusion likelihood is equal to or less than a predetermined value. (Configuration 3) 3. The information processing device according to configuration 2, wherein the determining means determines a display position of the label at a position where the label is included in the detection area. (Configuration 4) 4. The information processing device according to any one of configurations 1 to 3, further comprising a third acquisition means for acquiring a size of the label based on information to be displayed on the label. (Configuration 5) The information processing device according to any one of configurations 1 to 4, characterized in that the determination means determines the display position of the label based on the occlusion likelihood acquired by the second acquisition means and a display position score representing a priority of the display position of the label. (Configuration 6) 6. The information processing device according to configuration 5, wherein the display position score has a higher priority as the distance from another label whose display position has been determined increases. (Configuration 7) The information processing device described in any one of configurations 1 to 6, characterized in that the second acquisition means acquires the occlusion likelihood by reducing the size of the label and / or rotating the orientation of the label when the minimum value of the occlusion likelihood is not below a predetermined value. (Configuration 8) The information processing device according to any one of configurations 1 to 7, wherein the first acquisition means acquires a likelihood that the detection target exists for each cropped image obtained by cropping the image so as to include the detection area. (Configuration 9) 10. The information processing device according to any one of configurations 1 to 9, wherein the determining means determines a display position of the label based on a history of the occlusion likelihood that is stored. (Configuration 10) The information processing device according to any one of configurations 1 to 9, characterized in that the display means displays a polygon indicating a first detection area within the detection areas and a label belonging to the first detection area in the same color, and displays a polygon indicating a second detection area adjacent to the first detection area and a label belonging to the second detection area in a different color. (Configuration 11) 11. The information processing device according to configuration 10, wherein the display means displays the polygon indicating the detection area and the label belonging to the detection area using up to five different colors. (Configuration 12) The information processing device according to any one of configurations 1 to 11, characterized in that the display means highlights the label belonging to the detection area that has been selected by a user operation or the label that has been selected. (Configuration 13) 13. The information processing device according to any one of configurations 1 to 12, wherein the detection target is a vehicle. (Configuration 14) The first acquisition means further acquires a likelihood that a vehicle exists and a likelihood that a license plate of the vehicle exists for each of the regions of the image, The information processing device described in configuration 13, characterized in that the determination means determines the display position of the label based on the occlusion likelihood obtained based on the likelihood of the vehicle and the likelihood of the vehicle's license plate, and the size of the label. (Configuration 15) An imaging device that captures the image; 15. An information processing system comprising: an information processing device according to any one of configurations 1 to 14. (Method 1) A first acquisition step of acquiring a likelihood that a detection target exists for each of a plurality of regions of an image divided into the plurality of regions; a second acquisition step of acquiring an occlusion likelihood representing a likelihood of the detection target being occluded by the label for each of the regions of the image, based on a size of a label indicating information related to the detection region set for the image and the likelihood acquired in the first acquisition step; a determination step of determining a display position of the label in the image based on the occlusion likelihood acquired in the second acquisition step; a display step of displaying, on a display unit, the label corresponding to the detection area at the display position of the label determined in the determination step. (Program 1) A first acquisition step of acquiring a likelihood that a detection target exists for each of a plurality of regions of an image divided into the plurality of regions; a second acquisition step of acquiring an occlusion likelihood representing a likelihood of the detection target being occluded by the label for each of the regions of the image, based on a size of a label indicating information related to a detection region set for the image and the likelihood acquired in the first acquisition step; a determination step of determining a display position of the label in the image based on the occlusion likelihood acquired in the second acquisition step; a display step of displaying, on a display unit, the label corresponding to the detection area at the display position of the label determined in the determination step. [Explanation of symbols]

[0075] 110a to 110d: Imaging device 130: Information processing device 201: Imaging unit 202: Image processing unit 203: Arithmetic processing unit 204: Distribution unit 301: Imaging control unit 302: Signal processing unit 303: Storage unit 304: Control unit 305: Analysis unit 306: Network communication unit 401: Processor 402: RAM 403: ROM 404: Storage unit 405: Input unit 406: Display unit 407: Communication I / F 501: Network communication unit 502: Control unit 503: Display unit 504: Operation unit 505: Analysis unit 506: Storage unit

Claims

1. a first acquisition means for acquiring a likelihood that a detection target exists for each of a plurality of regions of an image divided into the plurality of regions; a second acquisition means for acquiring an occlusion likelihood representing a likelihood of the detection target being occluded by the label for each of the regions of the image, based on a size of a label indicating information related to a detection region set for the image and the likelihood acquired by the first acquisition means; a determination means for determining a display position of the label in the image based on the occlusion likelihood acquired by the second acquisition means; a display unit that displays the label corresponding to the detection area at a display position of the label determined by the determination unit.

2. The information processing apparatus according to claim 1 , wherein the determining means determines the display position of the label at a position where the detection area and the label at least partially overlap and the occlusion likelihood is equal to or smaller than a predetermined value.

3. 3. The information processing apparatus according to claim 2, wherein the determining means determines the display position of the label to be a position where the label is included in the detection area.

4. 2. An information processing apparatus according to claim 1, further comprising a third acquisition means for acquiring a size of the label based on information to be displayed on the label.

5. The information processing apparatus according to claim 1 , wherein the determining means determines the display position of the label based on the occlusion likelihood acquired by the second acquiring means and a display position score representing a priority of the display position of the label.

6. The information processing apparatus according to claim 5 , wherein the display position score has a higher priority as the distance from another label whose display position has been determined increases.

7. The information processing device according to claim 1 , characterized in that, when the minimum value of the occlusion likelihood is not below a predetermined value, the second acquisition means acquires the occlusion likelihood by performing at least one of reducing a size of the label and rotating an orientation of the label.

8. The information processing apparatus according to claim 1 , wherein the first acquisition means acquires the likelihood that the detection target exists for each cropped image obtained by cropping the image so as to include the detection area.

9. The information processing apparatus according to claim 1 , wherein the determining means determines the display position of the label based on a history of the occlusion likelihood that is stored.

10. 2. The information processing device according to claim 1, characterized in that the display means displays a polygon indicating a first detection area within the detection areas and a label belonging to the first detection area in the same color, and displays a polygon indicating a second detection area adjacent to the first detection area and a label belonging to the second detection area in a different color.

11. 11. The information processing apparatus according to claim 10, wherein the display means displays the polygon indicating the detection area and the label belonging to the detection area using up to five different colors.

12. The information processing apparatus according to claim 1 , wherein the display means highlights the label belonging to the detection area that has been selected by a user operation or the label that has been selected.

13. 2. The information processing apparatus according to claim 1, wherein the detection target is a vehicle.

14. The first acquisition means further acquires a likelihood that a vehicle exists and a likelihood that a license plate of the vehicle exists for each of the regions of the image, The information processing device according to claim 13, characterized in that the determination means determines the display position of the label based on the occlusion likelihood obtained based on the likelihood of the vehicle and the likelihood of a license plate of the vehicle, and on a size of the label.

15. An imaging device that captures the image; 2. An information processing system comprising the information processing device according to claim 1.

16. a first acquisition step of acquiring a likelihood that a detection target exists for each of a plurality of regions of an image divided into the plurality of regions; a second acquisition step of acquiring an occlusion likelihood representing a likelihood of the detection target being occluded by the label for each of the regions of the image, based on a size of a label indicating information related to a detection region set for the image and the likelihood acquired in the first acquisition step; a determination step of determining a display position of the label in the image based on the occlusion likelihood acquired in the second acquisition step; a display step of displaying, on a display unit, the label corresponding to the detection area at the display position of the label determined in the determination step.

17. a first acquisition step of acquiring a likelihood that a detection target exists for each of a plurality of regions of an image divided into the plurality of regions; a second acquisition step of acquiring an occlusion likelihood representing a likelihood of the detection target being occluded by the label for each of the regions of the image, based on a size of a label indicating information related to a detection region set for the image and the likelihood acquired in the first acquisition step; a determination step of determining a display position of the label in the image based on the occlusion likelihood acquired in the second acquisition step; a display step of displaying, on a display unit, the label corresponding to the detection area at the display position of the label determined in the determination step.

Citation Information

Patent Citations

  • Label placement calculation device, label placement calculation method, and program

    JP7015879B1