Presence determination device
The system uses an imaging and processing unit to form recognition figures and assign them to detection areas based on shortest distances, addressing inaccuracies in conventional systems by accurately determining presence in specific areas.
Patent Information
- Application Number
- JP2024078548
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-05-14
- Publication Date
- 2025-11-27
AI Technical Summary
Conventional object recognition systems often erroneously determine the presence of a person in both adjacent detection areas when they are on the boundary, or fail to recognize individuals further back than the closest person to the camera, leading to inaccurate presence determinations.
The system employs an imaging unit and information processing unit to form recognition figures from captured images, determine judgment distances, and assign these figures to specific determination areas based on the shortest mutual distances, using machine learning to improve accuracy.
This approach significantly reduces erroneous determinations by accurately assigning individuals to the correct detection areas, even when multiple people are present along the camera's line of sight.
Smart Images
Figure 2025173133000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a presence determination device, and more particularly to a device for determining whether or not an object is present in a specific area. [Background technology]
[0002] Devices that recognize specific objects such as people and luggage from images captured by cameras are widely used. Such object recognition devices are used in security systems installed in buildings, systems that detect people at event venues, systems that manage the collection and delivery of luggage, and the like. Patent Document 1 listed below describes a stay status display system that applies an object recognition device. This stay status display system detects the presence of people in a target area such as an office, conference room, or common room based on camera images. Furthermore, stay information indicating whether or not a person is present in a detection area where people are expected to be present is acquired. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-181221 Summary of the Invention [Problem to be solved by the invention]
[0004] With conventional technology, if a person is present on the boundary between two adjacent detection areas, it may be erroneously determined that a person is present in both detection areas, or that a person is not present in either detection area. Also, if there are multiple people along the camera's line of sight, only the person closest to the camera may be recognized, making it difficult to recognize people further back than the person closest to the camera.
[0005] The present invention aims to reduce the frequency of erroneous determinations in a device that determines whether an object is present in a specific area. [Means for solving the problem]
[0006] The present invention comprises an imaging unit that generates captured image data, and an information processing unit, and the information processing unit executes the following steps: an object recognition process that forms a recognition figure representing an area covered by an object based on the captured image data; a distance determination process that, when a reference point of the recognition figure formed for the object is included in a judgment area set on an image represented by the captured image data, determines a judgment distance between the reference point of the recognition figure and a judgment point in the judgment area; and an assignment process that, from among a plurality of pairs of the recognition figure and the judgment area, determines an assignment determination pair that assigns one recognition figure to one judgment area from pairs for which the judgment distance has been determined, based on the judgment distance.
[0007] Preferably, the allocation process includes an allocation confirmation process that focuses on each of the recognition figures to identify one of the judgment areas with the shortest judgment distance, focuses on each of the judgment areas to identify one of the recognition figures with the shortest judgment distance, and identifies as the allocation confirmation pair a pair of the recognition figure and the judgment area that has been identified as having the shortest mutual judgment distance.
[0008] Preferably, the allocation process is characterized by including a process of repeatedly executing the allocation confirmation process and excluding the pair of the recognition figure and the judgment area identified as the allocation confirmation pair in the previously executed allocation confirmation process from the target of the next allocation confirmation process to be executed.
[0009] Preferably, the object is a person, and the information processing unit constructs a machine learning model by acquiring machine learning data that corresponds the captured image data for machine learning with information indicating the state of the person in the captured image for machine learning shown by the captured image data for machine learning, for multiple cases where the state of the person in the captured image for machine learning is different, and by fitting the captured image data that is the subject of the object recognition processing into the machine learning model, recognizes the person in the image shown by the captured image data and forms the recognition figure for that person.
[0010] Preferably, the information processing section executes a process of setting each of the plurality of determination areas based on the captured image data.
[0011] Preferably, the information processing unit constructs a machine learning model by acquiring machine learning data that corresponds the captured image data for machine learning, information indicating the state of the specified object in the captured image for machine learning shown by the captured image data for machine learning, and information indicating the judgment area for the specified object, for multiple cases where the state of the specified object in the captured image for machine learning is different, and by fitting the captured image data that is the subject of the object recognition processing into the machine learning model, the specified object in the image shown by the captured image data is recognized and the judgment area for the specified object is set.
[0012] Preferably, the determination area is set for a seat that appears on the image represented by the captured image data. [Effects of the Invention]
[0013] According to the present invention, it is possible to reduce the frequency of erroneous determinations in a device that determines whether or not an object is present in a specific area. [Brief explanation of the drawings]
[0014] [Figure 1] FIG. 2 is a diagram illustrating the configuration of a presence determination device. [Figure 2] FIG. 10 is a diagram showing an example of a recognition pattern. [Figure 3] FIG. 10 is a diagram showing a plurality of seats arranged in a target area and a determination area set for each seat. [Figure 4] FIG. 10 is a diagram showing a determination distance table. [Figure 5] FIG. 10 is a diagram showing a determination distance table. [Figure 6] FIG. 10 is a diagram showing a determination distance table. [Figure 7] FIG. 10 is a diagram showing a determination distance table. [Figure 8] FIG. 1 shows a situation in which two seats are occupied by people. [Figure 9] This is a diagram of FIG. 8 with the human figures removed. DETAILED DESCRIPTION OF THE INVENTION
[0015] An embodiment of the present invention will be described with reference to the drawings. Identical components shown in multiple drawings will be assigned the same reference numerals, and their description will be simplified. FIG. 1 shows the configuration of a presence determination device 100 according to an embodiment of the present invention. The presence determination device 100 includes an imaging unit 10, an information processing unit 12, and a display 22. In the presence determination device 100, the imaging unit 10 captures an image, and the information processing unit 12 recognizes a person as an object in the captured image, and performs processing to identify one of multiple determination areas set on the image by the user in which the recognized person is present as a presence area.
[0016] The information processing unit 12 includes an object recognition unit 14, a model management unit 16, a presence determination unit 18, and a display processing unit 20. The information processing unit 12 may be configured with one or more processors. The processors execute programs to realize the functions of the object recognition unit 14, the model management unit 16, the presence determination unit 18, and the display processing unit 20.
[0017] The imaging unit 10 is fixed at a position where it captures an image of a target area 24 where a person to be detected as present is present. The imaging unit 10 captures an image of the target area 24, sequentially generates captured image data over time, and outputs the captured image data to the object recognition unit 14 and the display processing unit 20 at a predetermined frame rate. Here, a frame is a unit indicating the number of captured images, and the frame rate is a unit indicating the number of frames of captured image data output by the imaging unit 10 per unit time. The display processing unit 20 may generate a video signal based on the captured image data sequentially generated over time, and output the video signal to the display 22. In this case, the display 22 sequentially displays the captured images based on the video signal over time.
[0018] The object recognition unit 14 performs object recognition processing on the captured image data output from the imaging unit 10 and recognizes people in the captured image. The object recognition processing may be performed based on a machine learning model constructed by machine learning. For example, the object recognition unit 14 acquires machine learning data that associates the captured image data for machine learning with information indicating the state of a person in the captured image for machine learning indicated by the captured image data for machine learning, for multiple cases where the state of a person appearing in the captured image for machine learning is different, and outputs the machine learning data to the model management unit 16. Here, the state of a person includes the person's position, posture, etc. The object recognition unit 14 constructs a machine learning model based on the machine learning data stored in the model management unit 16. The object recognition unit 14 fits the captured image data into the machine learning model to identify the area occupied by a person in the captured image and recognize the person in the captured image.
[0019] Furthermore, the object recognition process may be performed based on a correlation calculation between the captured image and template image data stored in advance in the object recognition unit 14. For example, the object recognition unit 14 performs a correlation calculation between the captured image indicated by the captured image data and the template image indicated by the template image data, while changing the positional relationship between the captured image and the template image and the scaling ratio of the template image. Here, the template image is an image depicting a person. The object recognition unit 14 recognizes the person in the captured image based on the positional relationship between the captured image and the template image and the scaling ratio of the template image when the correlation value obtained by the correlation calculation exceeds a predetermined threshold and becomes maximum. The object recognition unit 14 may perform a correlation calculation on multiple template images in which the person appears in different poses, thereby recognizing people in various poses.
[0020] The object recognition unit 14 forms a recognition figure that indicates the range of the image of a person recognized on the captured image. That is, the object recognition unit 14 generates recognition image data that indicates the recognition figure. The object recognition unit 14 sequentially performs object recognition processing on each piece of captured image data that is sequentially output over time from the imaging unit 10, and generates recognition figure data from each piece of captured image data based on the captured image data. The object recognition unit 14 sequentially outputs the recognition figure data that is sequentially generated over time to the presence determination unit 18.
[0021] The presence determination unit 18 includes a determination area setting unit 26 and an area identification unit 28. When the target area 24 captured by the imaging unit 10 is displayed on the display 22, the determination area setting unit 26 sets a determination area for the captured image in response to a user operation. Here, the determination area is an area that is the target for determining whether or not a person is present. The determination area includes, for example, an area where seats, desks, etc. are arranged. In this embodiment, multiple determination areas are set on the captured image. The determination area in this embodiment is an area that includes seats that appear in the captured image.
[0022] FIG. 2 shows, as an example of the recognition figure 30, a rectangle that encloses the area covered by the image of a person 32. In this case, the recognition figure data may include position coordinate values of a pair of diagonal corners among the four corners. In FIG. 2, an xy coordinate plane is defined with the right direction of the captured image as the positive x-axis direction and the downward direction of the captured image as the positive y-axis direction. The recognition figure data includes position coordinate values (xmin, ymin) of the upper left corner and position coordinate values (xmax, ymax) of the lower right corner. The recognition figure data may also include position coordinate values (xmin, ymax) of the lower left corner and position coordinate values (xmax, ymin) of the upper right corner. Here, xmax is the maximum x-axis coordinate value of the area covered by the rectangular recognition figure, and xmin is the minimum x-axis coordinate value of the area covered by the rectangular recognition figure. Furthermore, ymax is the maximum y-axis coordinate value of the area covered by the rectangular recognition figure, and ymin is the minimum y-axis coordinate value of the area covered by the rectangular recognition figure.
[0023] A reference point Ps is defined for the recognition figure 30. In this embodiment, the upper left corner of the rectangular recognition figure 30 is set as the reference point Ps. The reference point Ps may be another corner of the recognition figure 30 or the center of gravity of the recognition figure 30. Note that the recognition figure 30 may be a rectangle, another polygon, a circle, an ellipse, or a substantially ellipse with a modified curvature of the ellipse.
[0024] FIG. 3 shows seats 40-1 to 40-3 arranged in the target area 24 and rectangular determination areas 42-1 to 42-3 set for each of the seats 40-1 to 40-3. In this embodiment, one person can sit in each of the seats 40-1 to 40-3. In this embodiment, the determination areas 42-1 to 42-3 are rectangular. Therefore, the area covered by each determination area 42-1 to 42-3 is specified by a pair of diagonal position coordinate values (umin, vmin) and (umax, vmax) on the xy plane. Here, umax is the maximum x-axis coordinate value of the area covered by the rectangle representing the determination area, and umin is the minimum x-axis coordinate value of the area covered by the rectangle representing the determination area. Furthermore, vmax is the maximum y-axis coordinate value of the area covered by the rectangle representing the determination area, and vmin is the minimum y-axis coordinate value of the area covered by the rectangle representing the determination area.
[0025] Judgment points Pd1 to Pd3 are defined in judgment regions 42-1 to 42-3, respectively. Judgment points Pd1 to Pd3 may be located, for example, at positions shifted in the positive x-axis direction from the upper left corner of the judgment region by α times the width of the judgment region in the x-axis direction, and then shifted in the positive y-axis direction by β times the width of the judgment region in the y-axis direction. Here, adjustment values α and β are numbers between 0 and 1. In this case, the position coordinate values of the judgment points are expressed as [umin+α(umax-umin),vmin+β(vmax-vmin)].
[0026] The adjustment values α and β may be determined by a user operation. The adjustment values α and β may be set to values that allow the region identification unit 28, which will be described later, to appropriately determine whether or not a person is present in the determination region. The adjustment values α and β may be different values for each of the determination regions 42-1 to 42-3.
[0027] The determination area may be a rectangle, or may be another polygon, a circle, an ellipse, or a roughly ellipse with a modified curvature. In this case, the user may set the determination point at a position that allows an appropriate determination of whether or not a person is present in the determination area.
[0028] 1 identifies a determination area that includes the reference point Psp of the recognition figure 30-p from among multiple determination areas 42-q (q=1 to n) on the captured image. Here, n indicates the number of determination areas set on the captured image. If the number of people recognized in the target area 24 is M, p is an integer between 1 and M and indicates the number assigned to the person recognized in the target area 24.
[0029] The region specifying unit 28 calculates a judgment distance dpq between the reference point Psp and the judgment point Pdq in the judgment region 42-q that includes the reference point Psp. The region specifying unit 28 performs this distance determination process for each judgment region 42-q that includes the reference point Psp.
[0030] 4 shows a determination distance table for each pair of one of the recognition figures 30-1 to 30-4 and one of the determination regions 42-1 to 42-5. That is, the table shows the determination distances dpq obtained by the distance determination process when the number of people recognized in the target region 24 is four and there are five determination regions 42-1 to 42-5.
[0031] In the example shown in FIG. 4, reference point Ps1 of recognition figure 30-1 is included in judgment regions 42-1 to 42-4 but is not included in judgment region 42-5. Reference point Ps2 of recognition figure 30-2 is included in judgment regions 42-1 to 42-3 and 42-5 but is not included in judgment region 42-4. Reference point Ps3 of recognition figure 30-3 is included in judgment regions 42-1, 42-4 and 42-5 but is not included in judgment regions 42-2 and 42-3. Reference point Ps4 of recognition figure 30-4 is included in judgment regions 42-1 to 42-3 but is not included in judgment regions 42-4 and 42-5. A judgment distance dpq is indicated for judgment region 42-q that includes reference point Psp of recognition figure 30-p, and the symbol "--" is indicated for judgment region 42-q that does not include reference point Psp of recognition figure 30-p.
[0032] The region specifying unit 28 focuses on each recognition pattern 30-p and specifies a judgment region 42-q with the shortest judgment distance. That is, the region specifying unit 28 specifies a judgment region 42-q with the shortest judgment distance dpq for the reference point Psp of each recognition pattern 30-p. In FIG. 5(a), the judgment region 42-q specified for each recognition pattern 30-p is shown by a horizontally shaded box.
[0033] Next, the region specifying unit 28 focuses on each judgment region 42-q and specifies the recognition figure 30-p with the shortest judgment distance. That is, for each judgment region 42-q, the region specifying unit 28 specifies the recognition figure 30-p corresponding to the reference point Psp with the shortest judgment distance dpq. In FIG. 5(b), the recognition figure 30-p specified for each judgment region 42-q in this manner is shown by a grayed-out box.
[0034] Furthermore, the region identification unit 28 identifies, as a confirmed allocation pair, a pair of the recognition graphic 30-p and the judgment region 42-q that has the shortest mutual judgment distance. That is, the region identification unit 28 identifies, as a confirmed allocation pair, a pair of the recognition graphic 30-p and the judgment region 42-q that corresponds to the horizontally shaded cells in FIG. 5(a) and the grayed-out cells in FIG. 5(b). In the example shown in FIGS. 5(a) and 5(b), the region identification unit 28 identifies, as a confirmed allocation pair, a pair of the recognition graphic 30-1 and the judgment region 42-2, and a pair of the recognition graphic 30-3 and the judgment region 42-5, as shown in FIG. 5(c).
[0035] The region identification unit 28 then performs a similar allocation confirmation process after excluding the assignment confirmation pair from the combination of the recognition graphic 30-p and the judgment region 42-q. In the above example, the recognition graphic 30-1 and 30-3 and the judgment region 42-2 and 42-5 are excluded, and the region identification unit 28 then performs a second allocation confirmation process. Figures 6(a) and 6(b) show the judgment distance table from which the correspondence between the recognition graphic 30-1 and 30-3 and the judgment region 42-2 and 42-5 has been excluded. The excluded correspondences are indicated by dark gray columns.
[0036] 6(a), the region specifying unit 28 focuses on the recognition graphics 30-2 and 30-4 and specifies a judgment region 42-q having the shortest judgment distance. That is, the region specifying unit 28 specifies a judgment region 42-3 having the shortest judgment distance for the recognition graphic 30-2. The region specifying unit 28 also specifies a judgment region 42-3 having the shortest judgment distance for the recognition graphic 30-4.
[0037] 6(b), the region specifying unit 28 focuses on the determination regions 42-1 and 42-3 and specifies the recognition pattern 30-p with the shortest determination distance. That is, the region specifying unit 28 specifies the recognition pattern 30-4 with the shortest determination distance for the determination region 42-1. The region specifying unit 28 also specifies the recognition pattern 30-2 with the shortest determination distance for the determination region 42-3.
[0038] The region specifying unit 28 specifies, as a confirmed allocation pair, a pair of the recognition graphic 30-p and the judgment region 42-q that has been specified as having the shortest mutual judgment distance. That is, as shown in FIG. 6(c), the region specifying unit 28 specifies the recognition graphic 30-2 and the judgment region 42-3 as a confirmed allocation pair.
[0039] The area identification unit 28 excludes the assignment confirmation pairs identified by the first and second assignment confirmation processes from the pair of recognition figure 30-p and judgment area 42-q, and then further performs a similar assignment confirmation process.
[0040] 7(a) and 7(b) show the judgment distance table in which the assignment confirmation pairs identified by the first and second assignment confirmation processes have been excluded. In the third assignment confirmation process, the area identification unit 28 identifies the pair of the recognition figure 30-4 and the judgment area 42-1 as the assignment confirmation pair.
[0041] As shown in Figure 7(c), the area identification unit 28 performs the allocation confirmation process three times to identify the pair of recognition figure 30-1 and judgment area 42-2, the pair of recognition figure 30-3 and judgment area 42-5, the pair of recognition figure 30-2 and judgment area 42-3, and the pair of recognition figure 30-4 and judgment area 42-1 as allocation confirmation pairs.
[0042] As a result, the region identification unit 28 recognizes that a person corresponding to the recognition graphic 30-4 is present in the determination region 42-1, and a person corresponding to the recognition graphic 30-1 is present in the determination region 42-2. The region identification unit 28 also recognizes that a person corresponding to the recognition graphic 30-2 is present in the determination region 42-3, and a person corresponding to the recognition graphic 30-3 is present in the determination region 42-5.
[0043] In this way, the area specifying unit 28 repeatedly executes the assignment determination process until all assignment determination pairs are specified. The pairs of recognition figures and judgment areas specified in the previous assignment determination process are excluded from the targets of the next assignment determination process.
[0044] Fig. 8 shows a situation in which persons 32A and 32B are seated in seats 40-2 and 40-3, respectively. Recognition figures 30-1 and 30-2 are formed for persons 32A and 32B. Determination areas 42-1 to 42-3 are set for seats 40-1 to 40-3. Fig. 9 shows a diagram in which the figures of persons 32A and 32B have been removed from Fig. 8 to make it easier to understand the geometric relationship between the recognition figures 30-1 and 30-2 and the determination areas 42-1 to 42-3.
[0045] The region specifying unit 28 specifies the determination regions 42-1 and 42-2 among the plurality of determination regions 42-1 to 42-3 on the captured image as determination regions that include the reference point Ps1 of the recognition figure 30-1. In response to this, the region specifying unit 28 calculates the determination distance d11 between the reference point Ps1 and the determination point Pd1, and further calculates the determination distance d12 between the reference point Ps1 and the determination point Pd2.
[0046] Furthermore, the region specifying unit 28 specifies the determination regions 42-2 and 42-3 among the plurality of determination regions 42-1 to 42-3 on the captured image as determination regions that include the reference point Ps2 of the recognition figure 30-2, and calculates the determination distance d22 between the reference point Ps2 and the determination point Pd2. In response to this, the region specifying unit 28 calculates the determination distance d22 between the reference point Ps2 and the determination point Pd2, and further calculates the determination distance d23 between the reference point Ps2 and the determination point Pd3.
[0047] The region specifying unit 28 focuses on the recognition graphic 30-1 and specifies a judgment region 42-2 as the judgment region having the shortest judgment distance, and also focuses on the recognition graphic 30-2 and specifies a judgment region 42-3 as the judgment region having the shortest judgment distance.
[0048] The region specifying unit 28 focuses on the judgment region 42-1 and identifies the recognition pattern 30-1 as the recognition pattern with the shortest judgment distance. The region specifying unit 28 also focuses on the judgment region 42-2 and identifies the recognition pattern 30-1 as the recognition pattern with the shortest judgment distance. The region specifying unit 28 also focuses on the judgment region 42-3 and identifies the recognition pattern 30-2 as the recognition pattern with the shortest judgment distance.
[0049] The area identification unit 28 identifies, as assignment confirmed pairs, the pair of recognition figure 30-1 and judgment area 42-2 and the pair of recognition figure 30-2 and judgment area 42-3, which are identified as having the shortest mutual judgment distance among the pairs of recognition figure and judgment area.
[0050] That is, the area specifying unit 28 recognizes that the person 32A corresponding to the recognition graphic 30-1 is present in the determination area 42-2, and the person 32B corresponding to the recognition graphic 30-2 is present in the determination area 42-3.
[0051] In this way, by performing a single allocation determination process, the determination areas 42-2 and 42-2 that form the allocation determination pairs are identified for all the recognition figures 30-1 and 30-2. Therefore, the area identification unit 28 ends the overall allocation determination process (allocation process) by completing the first allocation determination process.
[0052] Returning to FIG. 1, the presence determination unit 18 outputs information identifying the recognition graphics 30-1, 30-2, determination areas 42-1 to 42-3, and the presence areas for each recognition graphic, etc., to the display processing unit 20. The display processing unit 20 generates determination result image data showing an image in which images indicating the recognition graphics 30-1, 30-2, determination areas 42-1 to 42-3, and the confirmed allocation pairs, etc., are superimposed on the captured image. The display processing unit 20 converts the determination result image data into a video signal and outputs it to the display 22. The display 22 displays the determination result image based on the video signal. The determination result image may be, for example, an image in which the rectangles indicating the determination areas 42-2 and 42-3 in FIG. 8 are emphasized with a thick line or color, etc., and which indicates that the pair of the recognition graphic 30-1 and the determination area 42-2 and the pair of the recognition graphic 30-2 and the determination area 42-3 are confirmed allocation pairs.
[0053] The display processing unit 20 may also display on the display 22 seating status information for each date and time in the past. The seating status information may, for example, be information indicating whether or not a person was seated in a seat corresponding to a certain determination area at a certain date and time, using text or graphics. In addition to displaying information on the display 22, the display processing unit 20 may also execute a process to provide information to an external computer via communication. In this case, the display processing unit 20 may provide an application for displaying information to the external computer. The computer may display the information transmitted from the display processing unit 20 by executing the application.
[0054] The assignment process is reduced to a linear assignment process that assigns a recognition pattern to each judgment area. In this case, the area identification unit 28 calculates, for each judgment area, the evaluation distance between the judgment point of one judgment area and the reference point of the recognition pattern provisionally identified as forming a fixed assignment pair. Then, while changing the recognition pattern provisionally identified for each judgment area, the cost is calculated by adding up the evaluation distances calculated for all recognition areas, and a recognition pattern forming a fixed assignment pair for each judgment area is identified so as to minimize the cost.
[0055] In the presence determination device 100, the allocation determination process is repeatedly executed until all allocation determination pairs are identified. Pairs of recognition figures and determination areas identified in a previously executed allocation determination process are excluded from the targets of the next execution of the allocation determination process. This reduces the frequency of erroneous determinations that a person is present in both of the two determination areas, or that a person is not present between adjacent seats when in fact there is a person there.
[0056] Furthermore, the combination of the recognition figure and the determination area identified by the previously executed allocation confirmation process is excluded from the targets of the next allocation confirmation process. This prevents only the recognition figure of the person closest to the image capture unit 10 from being the target of the allocation confirmation process, even if there are multiple people along the line of sight of the image capture unit 10, and prevents the allocation confirmation process from not being executed for the recognition figures of people further back than the person closest to the person.
[0057] In the above, an embodiment has been described in which the determination areas 42-1 to 42-n are set by a user operation. The determination areas 42-1 to 42-n may be set by machine learning. For example, the presence determination unit 18 acquires machine learning data in which image data for machine learning, information indicating the state of a seat (a defined object that defines the determination area) appearing in the captured image for machine learning, such as the position and posture, and information indicating the determination area for that seat are associated with each other, for multiple cases in which the state of the seat appearing in the captured image for machine learning is different. The presence determination unit 18 constructs a machine learning model based on the machine learning data. The presence determination unit 18 sets the determination areas 42-1 to 42-n by fitting the captured image data output from the imaging unit 10 to the machine learning model.
[0058] Furthermore, the process of setting the determination areas 42-1 to 42-n may be performed based on a correlation calculation between the captured image and template image data stored in advance in the presence determination unit 18. For example, the presence determination unit 18 performs a correlation calculation between each of the captured image indicated by the captured image data and the template image indicated by the template image data, while changing the positional relationship between the captured image and the template image and the scaling ratio of the template image. Here, the template image is an image depicting a seat. The presence determination unit 18 recognizes the seats in the captured image and sets each of the determination areas 42-1 to 42-n based on the positional relationship between the captured image and the template image and the scaling ratio of the template image when the correlation value obtained by the correlation calculation exceeds a predetermined threshold and becomes maximum.
[0059] The above description also illustrates a presence determination process in which an image is captured by the imaging unit 10, the information processing unit 12 recognizes a person as an object in the captured image, and identifies one of multiple determination areas in which the recognized person exists as a presence area. Instead of this process, the information processing unit 12 may execute a presence determination process in which an object other than a person is recognized in the captured image, and identifies one of multiple determination areas in which the recognized object exists as a presence area. An example of an object other than a person is a car. In this case, the target area 24 is a parking lot, and the determination area is a parking space for one car. Examples of objects include products in a factory and luggage at a luggage collection and delivery center. If the object is a product in a factory, the determination area is a product storage area partitioned for each product. If the object is luggage at a luggage collection and delivery center, the determination area is a luggage storage area partitioned for each luggage.
[0060] As described above, for each of the multiple determination regions 42-1 to 42-n, determination points Pd1 to Pdn are determined based on the adjustment values α and β. The positions of the determination points Pd1 to Pdn vary depending on the adjustment values α and β determined for each determination region, and the determination distance dpq (q = 1 to n) obtained for each recognition graphic 30-p also varies. The presence determination unit 18 may determine multiple determination points with different adjustment values α and β for each of the multiple determination regions 42-1 to 42-n. In this case, multiple determination distances dpq (q = 1 to n) with different adjustment values α and β are determined for one recognition graphic 30-p. The presence determination unit 18 may use a statistical value, such as the average or median, of the multiple determination distances dpq (q = 1 to n) with different adjustment values α and β obtained for one recognition graphic 30-p as the determination distance dpq (q = 1 to n) used for presence determination.
[0061] As described above, the imaging unit 10 outputs captured image data at a predetermined frame rate to the object recognition unit 14 and the display processing unit 20. The information processing unit 12 may execute the presence determination process for each frame. Alternatively, the information processing unit 12 may execute the presence determination process for each set of frames. In this case, the region identification unit 28 may use a statistical value, such as the average or median value of the determination distances dpq (q = 1 to n) calculated for each recognition figure 30-p over multiple frames, as the determination distance to be used for presence determination.
[0062] The frames to be subjected to the presence determination process may be frames acquired going back a predetermined number of frames, including the most recent frame. The presence area may be identified based on a moving average value of the determination distance dpq (q = 1 to n) calculated for each recognition pattern 30-p. This reduces the frequency of erroneous determinations that a person is not present in a seat even if the person leaves the seat for a moment, or that a person is present in a seat even if the person simply passes by the seat.
[0063] [Configuration of the present invention] Configuration 1: An imaging unit that generates captured image data and an information processing unit, The information processing unit an object recognition process for forming a recognition figure representing the range of an object based on the photographed image data; a distance determination process for determining a judgment distance between the reference point of the recognition figure formed for the object and a judgment point in the judgment area when the reference point of the recognition figure formed for the object is included in a judgment area set on the image represented by the photographed image data; an assignment process for identifying an assignment determination pair, which assigns one recognition pattern to one judgment region, from among a plurality of pairs of the recognition pattern and the judgment region for which the judgment distance has been calculated, based on the judgment distance; A presence determination device characterized by executing the above. Configuration 2: The presence determination device according to configuration 1, Focusing on each of the recognition figures, one of the judgment regions having the shortest judgment distance is identified; Focusing on each of the judgment regions, one of the recognition figures having the shortest judgment distance is identified; The presence determination device is characterized in that the allocation process includes an allocation confirmation process that identifies, as the allocation confirmation pair, a pair of the recognition pattern and the determination area that has been identified as having the shortest mutual determination distance. Configuration 3: The presence determination device according to configuration 1, The allocation process includes: The presence determination device is characterized by including a process of repeatedly executing the allocation confirmation process and excluding the pair of the recognition figure and the determination area identified as the allocation confirmation pair in the previously executed allocation confirmation process from the target of the next allocation confirmation process. Configuration 4: The presence determination device according to any one of configurations 1 to 3, the object is a person, The information processing unit constructing a machine learning model by acquiring machine learning data that associates the captured image data for machine learning with information indicating the state of the person in the captured image for machine learning indicated by the captured image data for machine learning, for multiple cases where the state of the person in the captured image for machine learning is different; A presence determination device characterized by recognizing a person in an image shown by the captured image data by fitting the captured image data that is the subject of the object recognition processing into the machine learning model, and forming the recognition figure for that person. Configuration 5: The presence determination device according to any one of configurations 1 to 4, The information processing unit The presence determination device executes a process of setting each of the plurality of determination areas based on the captured image data. Configuration 6: The presence determination device according to configuration 5, The information processing unit A machine learning model is constructed by acquiring machine learning data that associates the captured image data for machine learning, information indicating the state of a specified object in the captured image for machine learning indicated by the captured image data for machine learning, and information indicating the judgment area for the specified object in multiple cases where the state of the specified object in the captured image for machine learning is different; A presence determination device characterized by recognizing the specified object in the image shown by the captured image data by fitting the captured image data that is the subject of the object recognition processing to the machine learning model, and setting the determination area for the specified object. Configuration 7: The presence determination device according to any one of configurations 1 to 6, The presence determination device is characterized in that the determination area is set for a seat that appears on an image shown by the captured image data. [Explanation of symbols]
[0064] 10 imaging unit, 12 information processing unit, 14 object recognition unit, 16 model management unit, 18 presence determination unit, 20 display processing unit, 22 display, 24 target area, 26 determination area setting unit, 28 area identification unit, 30, 30-1, 30-2 recognized figures, 32, 32A, 32B person, 40-1 to 40-3 seat, 42-1 to 42-3 determination area, 100 presence determination device, Ps, Ps1, Ps2 reference points, Pd, Pd1 to Pd3 determination points.
Claims
1. An imaging unit that generates photographed image data and an information processing unit, The information processing unit an object recognition process for forming a recognition figure representing the range of an object based on the photographed image data; a distance determination process for determining a judgment distance between the reference point of the recognition figure formed for the object and a judgment point in the judgment area when the reference point of the recognition figure formed for the object is included in a judgment area set on the image represented by the photographed image data; an assignment process for identifying an assignment determination pair, which assigns one recognition pattern to one judgment region, from among a plurality of pairs of the recognition pattern and the judgment region for which the judgment distance has been calculated, based on the judgment distance; A presence determination device characterized by executing the above.
2. The presence determination device according to claim 1, Focusing on each of the recognition figures, one of the judgment regions having the shortest judgment distance is identified; Focusing on each of the judgment regions, one of the recognition figures having the shortest judgment distance is identified; The presence determination device is characterized in that the allocation process includes an allocation confirmation process that identifies, as the allocation confirmation pair, a pair of the recognition pattern and the determination area that has been identified as having the shortest mutual determination distance.
3. The presence determination device according to claim 2, The allocation process includes: The presence determination device is characterized by including a process of repeatedly executing the allocation confirmation process and excluding the pair of the recognition figure and the determination area identified as the allocation confirmation pair in the previously executed allocation confirmation process from the target of the next allocation confirmation process.
4. The presence determination device according to any one of claims 1 to 3, the object is a person, The information processing unit constructing a machine learning model by acquiring machine learning data that associates the captured image data for machine learning with information indicating the state of the person in the captured image for machine learning indicated by the captured image data for machine learning, for multiple cases where the state of the person in the captured image for machine learning is different; A presence determination device characterized by recognizing a person in an image shown by the captured image data by fitting the captured image data that is the subject of the object recognition processing into the machine learning model, and forming the recognition figure for that person.
5. The presence determination device according to any one of claims 1 to 3, The information processing unit The presence determination device executes a process of setting each of the plurality of determination areas based on the captured image data.
6. The presence determination device according to claim 5, The information processing unit A machine learning model is constructed by acquiring machine learning data that associates the captured image data for machine learning, information indicating the state of a specified object in the captured image for machine learning indicated by the captured image data for machine learning, and information indicating the judgment area for the specified object in multiple cases where the state of the specified object in the captured image for machine learning is different; A presence determination device characterized by recognizing the specified object in the image shown by the captured image data by fitting the captured image data that is the subject of the object recognition processing to the machine learning model, and setting the determination area for the specified object.
7. The presence determination device according to claim 5, The presence determination device is characterized in that the determination area is set for a seat that appears on an image shown by the captured image data.
Citation Information
Patent Citations
Stay status display system and stay status display method
JP2018181221A