Auxiliary control device, auxiliary control method and computer readable storage medium

Abstract

The invention provides an assist control device, an assist control method, and a program computer-readable storage medium. The assist control device is provided with: a first acquisition unit that acquires position information of a first moving object detected from an image captured by an imaging device; a second acquisition unit that acquires position information of the second moving objects measured in each of the second moving objects; a recognition processing unit that performs a recognition process in which the position information of the first moving object and the position information of the second moving object are compared for each combination of the first moving object and the second moving object, and a combination of moving objects in which the degree of coincidence of the position information is higher than a predetermined value is recognized as one moving object; and an assistance control unit that performs control for traffic assistance with respect to the moving object on the basis of the position information of the moving object, and the recognition processing unit limits the position information to position information within a predetermined range set with the position of the imaging device as a reference.

Description

Technical Field

[0001] The invention relates to an auxiliary control device, an auxiliary control method and a computer-readable storage medium. Background Art

[0002] In recent years, efforts to provide access to sustainable transportation systems for disadvantaged persons among traffic participants have become increasingly active. To achieve this goal, research and development efforts have been devoted to further improving the safety and convenience of traffic through research and development related to preventive safety technologies. Patent documents 1 to 6 describe technologies related to vehicle operation assistance.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2019-175201

[0004] Patent Document 2: Japanese Patent Application Publication No. 2021-92840

[0005] Patent Document 3: Japanese Patent Application Publication No. 2019-78554

[0006] Patent Document 4: International Publication No. 2020 / 165981

[0007] Patent Document 5: International Publication No. 2019 / 188429

[0008] Patent Document 6: Japanese Patent Application Publication No. 2021-167770 Summary of the invention

[0009] However, in terms of preventive safety technology, the increase in processing load for assisting has become a problem. In order to solve the above problem, the present application aims to reduce the processing load in the assisting control device. In addition, it also contributes to the development of a sustainable transportation system.

[0010] In a first aspect of the present invention, an auxiliary control device is provided. The auxiliary control device includes a first acquisition unit that acquires position information of a first mobile object detected from an image captured by a camera. The auxiliary control device includes a second acquisition unit that acquires position information of the second mobile object measured in each of the second mobile objects. The auxiliary control device includes a recognition processing unit that performs recognition processing in which the position information of the first mobile object and the position information of the second mobile object are compared for each combination of the first mobile object and the second mobile object, and a combination of mobile objects whose position information consistency is higher than a predetermined value is recognized as one mobile object. The auxiliary control device includes an auxiliary control unit that performs control for traffic assistance for mobile objects based on the position information of the mobile object obtained by applying the recognition processing to the position information of the first mobile object and the position information of the second mobile object. The recognition processing unit limits the position information of the first mobile object and the position information of the second mobile object that is the object of the recognition processing to position information within a predetermined range set with the position of the camera as a reference.

[0011] In the above-mentioned auxiliary control device, the predetermined range may be a predetermined fixed range including a position where the imaging device is installed.

[0012] In any of the above-mentioned auxiliary control devices, there may also be a range setting unit, which selects at least four position information having outermost coordinate components for two-axis coordinate axes determined within the predetermined range from the position information of the first moving object acquired by the first acquisition unit, and sets the range defined by the line passing through each of the selected position information as the predetermined range.

[0013] In any of the above-mentioned auxiliary control devices, the identification processing unit can compare the position represented by the position information of the first mobile object and the position represented by the position information of the second mobile object according to each combination of the first mobile object and the second mobile object, and identify the combination of mobile objects whose position consistency is higher than a predetermined value as one mobile object.

[0014] In any of the above-mentioned auxiliary control devices, the recognition processing unit may recognize a combination of moving objects whose degree of coincidence of the positions is higher than a predetermined value within a predetermined time as one moving object.

[0015] In any of the above-mentioned auxiliary control devices, the identification processing unit can also compare the moving direction determined based on the history of the position information of the first mobile object and the moving direction determined based on the history of the position information of the second mobile object for each combination of the first mobile object and the second mobile object, and identify the combination of mobile objects whose position consistency is higher than a predetermined value and whose moving direction consistency is higher than a predetermined value as one mobile object.

[0016] In any of the above-mentioned auxiliary control devices, the recognition processing unit can recognize a combination of moving objects whose position consistency is higher than a predetermined value within a predetermined time and whose moving direction consistency is higher than a predetermined value within a predetermined time as one moving object.

[0017] In any of the above-mentioned auxiliary control devices, it can be configured that the recognition processing unit performs the following processing for each combination of the first moving object and the second moving object: determines over time whether the consistency of the positions is higher than a predetermined value, and each time it is determined that the consistency of the positions is higher than the predetermined value, increases the evaluation value by a predetermined value, and each time it is determined that the consistency of the positions is below a predetermined value, decreases the evaluation value by a predetermined value, and the recognition processing unit performs recognition processing to identify the combination of moving objects whose evaluation values ​​reach the predetermined value as one moving object.

[0018] In any of the above-mentioned auxiliary control devices, the identification processing unit can perform the following processing for each combination of the first moving object and the second moving object: determine over time whether the consistency of the position is higher than a predetermined value and the consistency of the moving direction is higher than a predetermined value, and each time it is determined that the consistency of the position is higher than a predetermined value and the consistency of the moving direction is higher than a predetermined value, increase the evaluation value by a predetermined value, and each time it is determined that the consistency of the position is below a predetermined value or the consistency of the moving direction is below a predetermined value, decrease the evaluation value by a predetermined value, and perform identification processing to identify the combination of moving objects whose evaluation values ​​reach the predetermined value as one moving object.

[0019] In any of the above-mentioned auxiliary control devices, when it is judged that there are moving objects approaching each other within a predetermined time based on the position information of the moving object obtained by applying the recognition processing to the position information of the first moving object and the position information of the second moving object, the auxiliary control unit can perform control for sending an alarm to at least one of the moving objects that may approach each other.

[0020] In any of the above-mentioned auxiliary control devices, the auxiliary control unit can predict the position of a mobile object that moves from within the shooting range of the shooting device to outside the shooting range based on the history of the position information of the mobile object obtained by applying the recognition processing to the position information of the first mobile object and the position information of the second mobile object, and judge whether there are mobile objects that approach each other within a predetermined time based on the predicted position of the mobile object and the position indicated by the position information of other mobile objects.

[0021] In a second aspect of the present invention, an auxiliary control method is provided. The auxiliary control method has a step of acquiring position information of a first mobile object detected from an image captured by a camera. The auxiliary control method has a step of acquiring position information of the second mobile object measured in each of the second mobile objects. The auxiliary control method has a step of performing recognition processing, in which the position information of the first mobile object and the position information of the second mobile object are compared for each combination of the first mobile object and the second mobile object, and a combination of mobile objects whose position information consistency is higher than a predetermined value is recognized as one mobile object. The auxiliary control method has a step of performing control for traffic assistance for mobile objects based on the position information of the mobile object obtained by applying the recognition processing to the position information of the first mobile object and the position information of the second mobile object. In the step of performing the recognition processing, the position information of the first mobile object and the position information of the second mobile object that is the object of the recognition processing is limited to the position information within a predetermined range set based on the position of the camera.

[0022] In a third aspect of the present invention, there is provided a program that causes a computer to function as any one of the above-mentioned auxiliary control devices.

[0023] In addition, the above summary of the invention does not list all the features of the present invention. In addition, a sub-combination of these feature groups may also be an invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A usage scenario of the assistance system 10 is schematically shown.

[0025] Figure 2 The functional structure of the auxiliary device 60 is shown.

[0026] Figure 3 The first reception information received by the assist device 60 from the imaging device 70 is shown in a table format.

[0027] Figure 4 The second reception information received by the auxiliary device 60 from the user terminal 82 is shown in the form of a table.

[0028] Figure 5 It is a diagram for explaining the recognition processing in the recognition processing unit 240 .

[0029] Figure 6 It is another diagram for explaining the recognition processing in the recognition processing unit 240 .

[0030] Figure 7 Another example of the range to be processed for recognition is schematically shown.

[0031] Figure 8 This is a diagram for explaining an assisting method when the user 80 f moves out of the imaging range of the imaging device 70 .

[0032] Fig. 9 An example of a flowchart related to an assist control method performed by the assist device 60 is shown.

[0033] Fig.10 An example of a computer 2000 is shown. DETAILED DESCRIPTION

[0034] Hereinafter, the present invention will be described by way of the embodiments of the invention, but the following embodiments do not limit the claimed invention. In addition, not all combinations of features described in the embodiments are essential to the solution means of the invention.

[0035] Figure 1 The following schematically shows a usage scenario of the support system 10. The support system 10 includes vehicles 20a, 20b, 20c, and 20d, a camera 70, a support device 60, and user terminals 82a, 82b, 82e, and 82g.

[0036] In this embodiment, the vehicle 20a, the vehicle 20b, the vehicle 20c, and the vehicle 20d are sometimes collectively referred to as "the vehicle 20". The user terminal 82a, the user terminal 82b, the user terminal 82e, and the user terminal 82g are sometimes collectively referred to as "the user terminal 82". The user 80e, the user 80f, and the user 80g are sometimes collectively referred to as "the user 80".

[0037] In the present embodiment, the vehicle 20 is a vehicle that travels on the road 90. The vehicle 20 is an example of a moving object.

[0038] User terminal 82a and user terminal 82b are terminals held by passengers of vehicle 20a and vehicle 20b, respectively. User terminal 82e is a terminal held by user 80e. User terminal 82g is a terminal held by user 80g. User 80f does not hold a user terminal. In this embodiment, user 80e, user 80f, and user 80g are pedestrians.

[0039] The user terminal 82 is a portable terminal such as a smartphone. The user terminal 82 is an example of a mobile object. The user terminal 82 periodically transmits the current position information of the user terminal 82 detected by the position sensor including the GNSS receiver to the auxiliary device 60. For example, the user terminal 82 transmits the longitude and latitude information indicating the current position of the user terminal 82 to the auxiliary device 60.

[0040] The camera 70 is a camera installed on a traffic infrastructure, and acquires the positions of the vehicle 20 and the user 80 within the camera range of the camera 70 by analyzing the captured images, and transmits the acquired positions of the vehicle 20 and the user 80 to the assisting device 60 .

[0041] The vehicle 20 and the user 80 are traffic participants. The vehicle 20 and the user 80 are examples of mobile objects that can be assistance targets of the assistance device 60.

[0042] The support device 60 receives the position information transmitted from the user terminal 82 via mobile communication. The support device 60 may receive the information transmitted from the user terminal 82 and the camera 70 via any combination of mobile communication, the Internet, or a communication line such as a dedicated line.

[0043] exist Figure 1 In the illustrated situation, the vehicle 20a, the vehicle 20c, the vehicle 20d, and the user 80f are located within the shooting range of the camera 70. The camera 70 determines the respective positions of the vehicle 20a, the vehicle 20c, the vehicle 20d, and the user 80f by analyzing the images acquired by shooting, and transmits position information indicating the determined positions to the assisting device 60.

[0044] The assistance device 60 selects a moving object to be assisted from the vehicle 20 and the user 80 based on the respective position information of the user terminal 82 received from the user terminal 82 and the position information of the vehicle 20a, the vehicle 20c, the vehicle 20d and the user 80f received from the shooting device 70, and provides assistance to the selected moving object.

[0045] As an example, the assistance device 60 determines that the vehicle 20a and the user 80e are close to each other within a predetermined time based on the history of the position information of the vehicle 20a acquired by the camera 70 and the history of the position information of the user 80e detected by the user terminal 82e. In this case, the assistance device 60 assists the user 80e by sending assistance information instructing the user terminal 82e to output an alarm.

[0046] Here, the assistance device 60 receives not only the position information of the user terminal 82a from the user terminal 82a held by the passenger of the vehicle 20a, but also the position information of the vehicle 20a acquired by analyzing the image by the camera 70. The user terminal 82a moves together with the vehicle 20a. Therefore, it is not preferable to perform the assistance-related processing by assuming that there are moving objects at the position detected by the user terminal 82a and the position of the vehicle 20a acquired by the camera 70, but it is desirable to recognize the user terminal 82a and the vehicle 20a as one moving object.

[0047] Therefore, the assistance device 60 performs recognition processing to compare the position information of the user terminal 82a detected by the user terminal 82a with the position information of the vehicle 20a acquired by the camera 70, and recognizes the user terminal 82a and the vehicle 20a as one moving object based on the comparison result. The assistance device 60 performs assistance for the traffic participant based on the position information collected by applying the recognition processing.

[0048] When performing the recognition process, the auxiliary device 60 does not compare all combinations of the position information acquired by the camera 70 and the position information detected by the user terminal 82, but limits the position information to be compared to the position information within the range 100. In one embodiment, the range 100 is set within a predetermined range including the position where the camera 70 is installed. In one embodiment, the range 100 may be a range where the distance from the position where the camera 70 is installed is less than a predetermined set value. The set value may be, for example, 300 m.

[0049] According to the support system 10 , the position information to be processed for recognition can be limited, thereby reducing the processing load on the support device 60 .

[0050] Figure 2 The following is a functional configuration of the support device 60. The support device 60 includes a support control device 200, a communication device 290, and a storage device 280.

[0051] The communication device 290 is responsible for the communication between the user terminal 82 and the camera 70 and the auxiliary device 60, respectively, based on the control of the auxiliary control device 200. The auxiliary control device 200 is implemented by a circuit including, for example, a calculation processing device including a processor. The auxiliary control device 200 can be implemented by a microcomputer including a CPU, ROM, RAM, I / O, bus, etc. The storage device 280 is implemented by including a non-volatile storage medium. The auxiliary control device 200 performs processing using information stored in the storage device 280. The storage device 280 stores map information including information indicating the position of the road 90.

[0052] The auxiliary control device 200 includes an acquisition unit 202, an auxiliary control unit 260, a recognition processing unit 240, and a range setting unit 230. The acquisition unit 202 includes a first acquisition unit 210 and a second acquisition unit 220. Figure 2 The functional form of a part of the functional structure shown.

[0053] The first acquisition unit 210 acquires the position information of the first moving object detected from the image captured by the imaging device 70. The first acquisition unit 210 acquires the position information of the first moving object from the position information transmitted by the imaging device 70.

[0054] The second acquisition unit 220 acquires the position information of the second moving object measured for each second moving object. The second moving object in this embodiment is the user terminal 82. The second acquisition unit 220 acquires the position information of the user terminal 82 from the position information transmitted by the user terminal 82.

[0055] In the present embodiment, the first acquisition unit 210 acquires the position information of the first moving object detected from the image captured by the camera 70, and the second acquisition unit 220 acquires the position information of the user terminal 82 measured in each user terminal 82. However, in a mode where the vehicle 20 is provided with a camera that captures an image of the surroundings of the vehicle 20, the first acquisition unit 210 may acquire the position information of the first moving object detected from the image captured by the camera provided by the vehicle 20. When the vehicle 20 is provided with a position sensor including a GNSS receiver, the second acquisition unit 220 may acquire the position information of the vehicle 20 measured by the position sensor provided by the vehicle 20.

[0056] The recognition processing unit 240 performs recognition processing to compare the position information of the first mobile object and the position information of the second mobile object for each combination of the first mobile object and the second mobile object, and recognizes the combination of mobile objects whose position information has a consistency higher than a predetermined value as one mobile object. At this time, the recognition processing unit 240 limits the position information of the first mobile object and the second mobile object as the object of recognition processing to the position information within a predetermined range set based on the position of the camera 70. The auxiliary control unit 260 performs control for traffic assistance for the mobile object based on the position information of the mobile object obtained by applying the recognition processing to the position information of the first mobile object and the position information of the second mobile object.

[0057] The predetermined range may be a predetermined fixed range 100 including the setting position of the photographing device 70. The range setting unit 230 may select at least four pieces of position information having outermost coordinate components respectively about two axis coordinate axes set within the predetermined range from the position information of the first moving object acquired by the first acquisition unit 210, and set the range defined by a line passing through each of the selected pieces of position information as the predetermined range.

[0058] As an example, the recognition processing unit 240 compares the position indicated by the position information of the first mobile object with the position indicated by the position information of the second mobile object for each combination of the first mobile object and the second mobile object, and recognizes the combination of mobile objects whose position consistency is higher than a predetermined value as one mobile object. The recognition processing unit 240 may recognize the combination of mobile objects whose position consistency is higher than a predetermined value within a predetermined time as one mobile object.

[0059] The recognition processing unit 240 may also compare the moving direction determined based on the history of the position information of the first moving object and the moving direction determined based on the history of the position information of the second moving object for each combination of the first moving object and the second moving object, and recognize the combination of mobile objects whose position consistency is higher than a predetermined value and whose moving direction consistency is higher than a predetermined value as one mobile object. The recognition processing unit 240 may recognize the combination of mobile objects whose position consistency is higher than a predetermined value within a predetermined time and whose moving direction consistency is higher than a predetermined value within a predetermined time as one mobile object.

[0060] The recognition processing unit 240 may perform a first evaluation value calculation process for each combination of the first moving object and the second moving object. The first evaluation value calculation process includes the following processes: (i) determining whether the degree of consistency of the position is higher than a predetermined value over time, and (ii) increasing the evaluation value by a predetermined value each time the degree of consistency of the position is determined to be higher than the predetermined value, and decreasing the evaluation value by a predetermined value each time the degree of consistency of the position is determined to be lower than the predetermined value. The recognition processing unit 240 performs recognition processing for recognizing a combination of moving objects whose evaluation values ​​reach a predetermined value as one moving object.

[0061] The recognition processing unit 240 may perform a second evaluation value calculation process for each combination of the first moving object and the second moving object. The second evaluation value calculation process includes the following process: determining over time whether the consistency of the position is higher than a predetermined value and the consistency of the moving direction is higher than a predetermined value, and increasing the evaluation value by a predetermined value each time it is determined that the consistency of the position is higher than a predetermined value and the consistency of the moving direction is higher than a predetermined value, and decreasing the evaluation value by a predetermined value each time it is determined that the consistency of the position is lower than a predetermined value or the consistency of the moving direction is lower than a predetermined value. The recognition processing unit 240 performs recognition processing for recognizing a combination of moving objects whose evaluation values ​​reach a predetermined value as one moving object.

[0062] When it is determined that there are moving objects approaching each other within a predetermined time based on the position information of the moving objects obtained by applying recognition processing to the position information of the first moving object and the position information of the second moving object, the auxiliary control unit 260 performs control for sending an alarm to at least one of the moving objects that have the possibility of approaching each other.

[0063] The assist control unit 260 can predict the position of a mobile object that has moved from the shooting range of the shooting device 70 to the outside of the shooting range based on the history of the position information of the mobile object obtained by applying the recognition processing to the position information of the first mobile object and the position information of the second mobile object, and judge whether there are mobile objects that are close to each other within a predetermined time based on the predicted position of the mobile object and the position indicated by the position information of other mobile objects. Thus, even for a mobile object that has moved to the outside of the shooting range of the shooting device 70, assistance can be continued for a certain period of time.

[0064] Figure 3 The first reception information received by the assisting device 60 from the photographing device 70 is shown in a table format. The first reception information includes an ID, a time, a position, a type, an acquisition unit, and an acquisition source position as data items.

[0065] ID represents identification information of a mobile object. The photographing device 70 assigns identification information to a mobile object acquired by analyzing an image. The position information transmitted from the photographing device 70 contains identification information assigned by the photographing device 70 to the mobile object.

[0066] The time indicates the time when the position is acquired in the photographing device. The position indicates the position of the moving object. Specifically, the position indicates the position of the moving object acquired by analyzing the image by the photographing device 70. The position may be, for example, longitude and latitude information.

[0067] The type indicates the type of the moving object. The imaging device 70 determines the type of the moving object acquired by analyzing the image. The position information transmitted from the imaging device 70 includes the type of the moving object determined by the imaging device 70.

[0068] The acquisition means is information indicating a means for acquiring the position information. Figure 3 The “infrastructure camera” indicates that the unit for acquiring the position information is a photographing device 70 provided in the road infrastructure.

[0069] The acquisition source position refers to the position information of the photographing device 70 provided as the acquisition unit. The acquisition source position may be longitude and latitude information.

[0070] Assumptions Figure 3 The IDs "101", "102" and "103" in FIG. 1 are identification information of the vehicle 20a, the vehicle 20c and the vehicle 20d, respectively, assigned by the camera 70. It is assumed that the ID "104" is identification information of the user 80f assigned by the camera 70.

[0071] Figure 4 The second reception information received by the auxiliary device 60 from the user terminal 82 is shown in the form of a table. The second reception information contains an ID, a time, a position, and an acquisition unit as data items.

[0072] ID represents identification information of a mobile object. The position information transmitted from the user terminal 82 includes identification information fixedly assigned to the user terminal 82.

[0073] The time indicates the time when the position information is acquired at the user terminal 82. The position indicates the position of the mobile object. Specifically, the position indicates the position of the vehicle 20 detected by the user terminal 82. The position may be, for example, longitude and latitude information.

[0074] The acquisition unit is a unit for acquiring position information. Figure 4 The “smartphone” indicates that the unit for acquiring the location information is a smartphone.

[0075] Assumptions Figure 4 The IDs "301", "302", "303" and "304" in FIG. 8 are identification information of the user terminal 82a, the user terminal 82b, the user terminal 82e and the user terminal 82g, respectively.

[0076] As reference Figure 1 As described above, the recognition processing unit 240 only detects the moving objects whose position information exists in the range 100 as the objects of the recognition processing. Figure 3 and Figure 4 In the example of Figure 3 The four positions of ID "101", ID "102", ID "103" and ID "104" in correspond to Figure 4The combination of the three positions of ID "301", ID "302" and ID "303" in the image is used as the object of identification processing. That is, Figure 4 The position of ID "304" is not the object of the recognition process. This can reduce the amount of calculation for the recognition process.

[0077] Figure 5 It is a diagram for explaining the recognition processing in the recognition processing unit 240 . Figure 5 This shows a case where it is determined that the first moving object and the second moving object are recognized as one moving object.

[0078] Figure 5 FIG. 2 shows a case where the position of the first moving object and the position of the second moving object are compared five times at predetermined time intervals. Figure 5 In FIG. 5 , position 511, position 512, position 513, position 514, and position 515 represent the position of the first moving object. Position 521, position 522, position 523, position 524, and position 525 represent the position of the second moving object. Figure 5 In the figure, for each of the first moving object and the second moving object, an arrow is used to show the moving direction determined according to the history of multiple position information. Here, the moving direction is positive, indicating whether it is heading in the positive direction or the reverse direction, with the moving direction of each moving object as positive.

[0079] The initial value of the evaluation value for identifying the first moving object and the second moving object as one moving object is 0. At the first moment, the position of the first moving object is position 511, and the position of the second moving object is position 521. The recognition processing unit 240 determines whether the position 521 of the second moving object exists in a predetermined range A1 centered on the position 511 and has the same moving direction. The position 521 is in the range A1 and has the same moving direction. Therefore, the evaluation value is increased by 1. The evaluation value thus becomes 1.

[0080] The same process is performed at the second to fifth moments. In other words, at the second moment, since the position 522 of the second moving object exists in the predetermined range A2 centered on the position 512 of the first moving object and the moving direction is the same, the recognition processing unit 240 adds 1 to the evaluation value. Thus, the evaluation value is 2 at the second moment.

[0081] At the third time, the position 523 of the second moving object does not exist in the predetermined range A3 centered on the position 513 of the first moving object. Therefore, the recognition processing unit 240 decrements the evaluation value by 1. Thus, the evaluation value is 1 at the third time.

[0082] At the fourth moment, since the position 524 of the second moving object exists in the predetermined range A4 centered on the position 514 of the first moving object and the moving direction is the same, the recognition processing unit 240 adds 1 to the evaluation value. Therefore, the evaluation value at the fourth moment is 2. At the fifth moment, since the position 525 of the second moving object exists in the predetermined range A5 centered on the position 515 of the first moving object and the moving direction is the same, the recognition processing unit 240 adds 1 to the evaluation value. Therefore, the evaluation value at the fifth moment is 3.

[0083] When the evaluation value is equal to or greater than the first threshold value 3, the recognition processing unit 240 determines that the first moving object and the second moving object are recognized as one moving object.

[0084] Figure 6 It is another diagram for explaining the recognition processing in the recognition processing unit 240 . Figure 6 This shows a case where it is determined that the first moving object and the second moving object are not recognized as one moving object.

[0085] and Figure 5 same, Figure 6 The comparison between the position of the first moving object and the position of the second moving object is performed five times at predetermined time intervals. At the first moment, the position of the first moving object is position 511, and the position of the second moving object is not in range A1. Therefore, the recognition processing unit 240 reduces the evaluation value by 1. Thus, the evaluation value at the first moment is -1.

[0086] Likewise, at the second time, since the position 622 of the second moving object does not exist in the range A2 centered on the position 512 of the first moving object, the recognition processing unit 240 decrements the evaluation value by 1. Thus, the evaluation value at the second time is -2.

[0087] At the third time, the position 623 of the second moving object exists in the range A3 centered on the position 513 of the first moving object, and the moving direction is the same. Therefore, the recognition processing unit 240 adds 1 to the evaluation value. Thus, the evaluation value at the third time is -1.

[0088] At the fourth moment, since the position 624 of the second moving object does not exist in the range A4 centered on the position 514 of the first moving object, the recognition processing unit 240 reduces the evaluation value by 1. Thus, the evaluation value at the fourth moment is -2. At the fifth moment, since the position 625 of the second moving object does not exist in the range A5 centered on the position 515 of the first moving object, the recognition processing unit 240 reduces the evaluation value by 1. Thus, the evaluation value at the fifth moment is -3.

[0089] When the evaluation value becomes equal to or less than the second threshold value −3, the recognition processing unit 240 determines that the first moving object and the second moving object are recognized as one moving object.

[0090] Figure 7 Other examples of the range to be processed for recognition are schematically shown. Figure 7 In FIG. 1 , the mark ○ represents the position of the first moving object detected by the imaging device 70 , and the mark × represents the position of the second moving object detected by the user terminal 82 , which is shown in the xy coordinate system.

[0091] The range setting unit 230 determines P1 having the minimum x-coordinate value, P2 having the maximum x-coordinate value, P3 having the minimum y-coordinate value, and P4 having the maximum y-coordinate value from the position of the first moving object detected by the imaging device 70. The range setting unit 230 sets a range 700 defined by a line L1 passing through P1 and parallel to the y-axis, a line L2 passing through P2 and parallel to the y-axis, a line L3 passing through P3 and parallel to the x-axis, and a line L4 passing through P4 and parallel to the x-axis as a range to be processed for recognition.

[0092] By determining whether the x-coordinate of the position of the second moving object is between the x-coordinate of P1 and the x-coordinate of P2 and the y-coordinate of the position of the second moving object is between the y-coordinate of P3 and the y-coordinate of P4, it is possible to determine whether the coordinates of the second moving object are within the range 700. As described above, since it is possible to determine based on the magnitude relationship of the coordinate values, it is possible to reduce the amount of calculation related to the process for extracting the coordinates of the second moving object included in the range 700. In addition, the second moving object whose position is detected near the first moving object whose position can actually be acquired by the camera 70 can be set as the object of the recognition process. As a result, it is expected that the amount of calculation required for the recognition process can be further reduced.

[0093] Figure 8 This is a diagram for explaining an assisting method when the user 80 f moves out of the imaging range of the imaging device 70 .

[0094] The auxiliary control unit 260 holds the position P and the moving speed v of the user 80f before the user 80f moves out of the shooting range. The auxiliary control unit 260 estimates the position of the user 80f based on the position P and the moving speed v within a predetermined time from the time point when the user 80f moves out of the shooting range. For example, the auxiliary control unit 260 can estimate the position P′ after T seconds based on the position P and the moving speed v.

[0095] The auxiliary control unit 260 estimates the position Q of the vehicle 20b after T seconds based on the history of the position information of the user terminal 82b. When it is determined that the distance between the position P′ and Q is less than a predetermined value, the auxiliary control unit 260 determines to provide assistance to the user 80f and / or the vehicle 20b. For example, the auxiliary control unit 260 provides assistance by instructing the user terminal 82b to output an alarm. In this case, the auxiliary control unit 260 can reduce the intensity of the alarm. For example, when the user 80f is within the shooting range of the camera 70, an alarm of "pedestrians are nearby" is output, and when the user 80f moves outside the shooting range of the camera 70, an alarm of "pedestrians may be nearby" is output. In this way, assistance can be provided taking into account the possibility of reduced assistance accuracy.

[0096] Fig. 9 An example of a flowchart related to an assist control method performed by the assist device 60 is shown.

[0097] In S900, the range setting unit 230 sets a range for extracting position information to be processed as a recognition target. The range setting unit 230 may be as described in reference to Figure 1 As in the range 100 described above, the range is set based on the position of the imaging device 70. The installation position of the imaging device 70 is included in the position information transmitted from the imaging device 70. The range setting unit 230 may be as described in reference to Figure 7 As in the range 700 described above, a range based on the position detected by the imaging device 70 is set.

[0098] In S902 , the recognition processing unit 240 extracts position information of a second moving object to be recognized within the range set in S900 .

[0099] In S904 , the recognition processing section 240 executes a loop process of looping the processes from S906 to S910 for a combination of the position of the first moving object and the position of the second moving object extracted in S902 .

[0100] In S906, the recognition processing unit 240 calculates the evaluation value. For example, the recognition processing unit 240 calculates the evaluation value by referring to Figure 5 and Figure 6 The method described calculates the evaluation value. Figure 5 and Figure 6 , the case where the calculation of the evaluation value is terminated at 5 calculation steps is described, but an upper limit value may be set for the number of calculation steps for calculating the evaluation value. The recognition processing unit 240 may terminate the calculation of the evaluation value when the number of calculation steps for the evaluation value reaches the upper limit value.

[0101] In S908, it is determined whether the evaluation value calculated in S906 is greater than the first threshold. If it is determined that the evaluation value is greater than the first threshold, the process of S910 is executed. If it is determined that the evaluation value is less than the first threshold, the process enters the next step of the loop process. In S910, the recognition processing unit 240 determines that the first moving object and the second moving object are recognized as one moving object.

[0102] In S912, the assistance control unit 260 makes an assistance judgment on whether to perform assistance. For example, the assistance control unit 260 predicts the position of the future mobile object for each of the multiple mobile objects after the recognition processing of the recognition processing unit 240 is applied, and when it is predicted that it will be close to other mobile objects within a predetermined time, it is determined that assistance is performed on the mobile object and / or the other mobile objects. The assistance control unit 260 can determine whether it is close to other mobile objects based on whether the distance between the mobile objects predicted after the predetermined time is less than a predetermined value.

[0103] In the case where the first moving object and the second moving object are identified as one moving object for auxiliary judgment, when both the position detected by the shooting device 70 and the position detected by the user terminal 82 are detected, one of the positions can be used for auxiliary judgment, and the average value of the position detected by the shooting device 70 and the position detected by the user terminal 82 can be used for auxiliary judgment.

[0104] In S914, the assistance control unit 260 performs assistance on the mobile object according to the assistance judgment in S912. For example, the assistance control unit 260 controls the communication device 290 to send alarm information indicating the output of an alarm to at least one of the mobile object predicted to be close to the other mobile object in the judgment in S912 and the other mobile object. Figure 1 In the example of FIG. 2 , when the user terminal 82e is the assistance target, the assistance control unit 260 sends to the user terminal 82e an alarm message instructing the user terminal 82e to output the alarm message by sound and / or screen display. When the user terminal 82e receives the alarm message, the user 80e is notified of the presence of a vehicle approaching the user 80e through the HMI (Human Machine Interface) function of the user terminal 82e.

[0105] According to the above-described assistance system 10, the position information of the object of the recognition processing used to determine whether to recognize a moving object can be limited to a specific range. As a result, the processing load required for the recognition processing can be reduced. As a result, the sending of the alarm information can be prevented from being delayed too much.

[0106] Fig.10An example of a computer 2000 that can fully or partially embody multiple embodiments of the present invention is shown. The program installed in the computer 2000 can cause the computer 2000 to function as a device such as the auxiliary device 60 or each unit of the device involved in the embodiment, perform operations associated with the device or each unit of the device, and / or perform a process involved in the embodiment or steps of the process. Such a program can be executed by the CPU 2012 in order to cause the computer 2000 to perform the processing flow described in this specification and specific operations associated with some or all of the functional blocks in the block diagram.

[0107] The computer 2000 according to the present embodiment includes a CPU 2012 and a RAM 2014, which are connected to each other via a main controller 2010. The computer 2000 further includes a ROM 2026, a flash memory 2024, a communication interface 2022, and an input / output chip 2040. The ROM 2026, the flash memory 2024, the communication interface 2022, and the input / output chip 2040 are connected to the main controller 2010 via the input / output controller 2020.

[0108] The CPU 2012 operates according to the programs stored in the ROM 2026 and the RAM 2014 , thereby controlling each unit.

[0109] The communication interface 2022 communicates with other electronic devices via a network. The flash memory 2024 stores programs and data used by the CPU 2012 in the computer 2000. The ROM 2026 stores a startup program executed by the computer 2000 when activated, and / or a program dependent on the hardware of the computer 2000. The input / output chip 2040 can also connect various input / output units such as a keyboard, a mouse, and a monitor to the input / output controller 2020 via an input / output port such as a serial port, a parallel port, a keyboard port, a mouse port, a monitor port, a USB port, and an HDMI (registered trademark) port.

[0110] The program is provided via a computer-readable storage medium such as a CD-ROM, DVD-ROM, or a USB flash drive, or a network. RAM 2014, ROM 2026, or flash memory 2024 is an example of a computer-readable storage medium. The program is installed in flash memory 2024, RAM 2014, or ROM 2026 and executed by CPU 2012. The information processing described in these programs is read by computer 2000, and the cooperation between the program and the various types of hardware resources described above is realized. The device or method can be constructed by implementing the operation or processing of information in accordance with the use of computer 2000.

[0111] For example, when the computer 2000 communicates with an external device, the CPU 2012 may execute a communication program loaded into the RAM 2014, and based on the processing described in the communication program, instruct the communication interface 2022 to perform communication processing. Under the control of the CPU 2012, the communication interface 2022 reads the transmission data stored in the transmission buffer processing area provided in the RAM 2014 and the recording medium such as the flash memory 2024, transmits the read transmission data to the network, and writes the reception data received from the network to the reception buffer processing area provided on the recording medium.

[0112] In addition, the CPU 2012 can read all or a necessary part of a file or database stored in a recording medium such as the flash memory 2024 to the RAM 2014, and execute various processes on the data in the RAM 2014. The CPU 2012 then writes the processed data back to the recording medium.

[0113] Various types of information such as programs, data, tables, and databases can be saved to a recording medium and applied to information processing. CPU2012 can perform various operations, information processing, conditional judgment, conditional branching, unconditional branching, information retrieval / replacement, etc., which are specified by the instruction sequence of the program, on the data read from RAM2014, and write the results back to RAM2014. In addition, CPU2012 can retrieve information in files, databases, etc. in the recording medium. For example, when multiple items having attribute values ​​of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, CPU2012 can retrieve an item that specifies the attribute value of the first attribute and is consistent with the condition from the multiple items, read the attribute value of the second attribute stored in the item, and thereby obtain the attribute value of the second attribute associated with the first attribute that meets the pre-set condition.

[0114] The program or software module described above can be stored in a computer-readable storage medium on or near the computer 2000. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium. The program stored in the computer-readable storage medium can be provided to the computer 2000 via a network.

[0115] The program installed in the computer 2000 and causing the computer 2000 to function as the auxiliary device 60 can be run in the CPU 2012 or the like, thereby causing the computer 2000 to function as each unit of the auxiliary device 60 (e.g., the auxiliary control device 200, etc.). The information processing described in these programs is read into the computer 2000, thereby functioning as a specific unit in which software and the various hardware resources described above cooperate, that is, each unit of the auxiliary device 60. And, by using these specific units to realize the calculation or processing of information corresponding to the purpose of use of the computer 2000 in this embodiment, a unique auxiliary device 60 corresponding to the purpose of use is constructed.

[0116] Various embodiments are described with reference to block diagrams, etc. In the block diagram, each functional block may represent (1) a step of a process for performing an operation or (2) each unit of a device having the function of performing an operation. Specific steps and each unit may be implemented by a dedicated circuit, a programmable circuit supplied together with computer-readable instructions stored on a computer-readable storage medium, and / or a processor supplied together with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuit may include digital and / or analog hardware circuits, and may also include integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include reconfigurable hardware circuits including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc., including memory elements.

[0117] The computer-readable storage medium may include any tangible device capable of storing instructions to be executed by an appropriate device, and as a result, the computer-readable storage medium having the instructions stored therein constitutes at least a part of a product containing instructions executable to implement a unit for performing the operations specified in the process flow or block diagram. As examples of computer-readable storage media, electrical storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. may be included. As more specific examples of computer-readable storage media, floppy disks (registered trademark), floppy disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), electrically erasable programmable read-only memories (EEPROM), static random access memories (SRAM), compact disk read-only memories (CD-ROM), digital versatile disks (DVD), Blu-ray (registered trademark) optical disks, memory sticks, integrated circuit cards, etc. may be included.

[0118] Computer-readable instructions may include assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine delegate instructions, microcode, firmware instructions, state setting data, or any source code or object code described in any combination of one or more programming languages ​​including object-oriented programming languages ​​such as Smalltalk (registered trademark), JAVA (registered trademark), C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0119] Computer readable instructions are provided to a processor or programmable circuit of a general-purpose computer, a special-purpose computer, or other programmable data processing device via a local or local area network (LAN), a wide area network (WAN) such as the Internet, and the computer readable instructions can be executed to implement a unit for performing the operations specified in the described processing flow or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0120] The present invention has been described above using the embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. It is obvious from the description of the claims that such changes or improvements can also be included in the technical scope of the present invention.

[0121] Regarding the execution order of each process of actions, processes, steps, and steps in the devices, systems, programs, and methods shown in the claims, specifications, and drawings, it should be noted that there is no special indication of "before", "preceding", etc., and in addition, as long as the output of the previous process is not used in the subsequent process, it can be implemented in any order. For the action flow in the claims, specifications, and drawings, even if "first", "next", etc. are used for convenience, it does not mean that it must be implemented in such an order.

[0122] [Description of Reference Numerals]

[0123] 10Assistance Systems

[0124] 20 vehicles

[0125] 60 auxiliary devices

[0126] 70 Camera

[0127] 80 users

[0128] 82 user terminals

[0129] 90 Road

[0130] 100 range

[0131] 200 Auxiliary Control Devices

[0132] 202 Acquisition Department

[0133] 210 First Acquisition Department

[0134] 220 Second Acquisition Unit

[0135] 230 Range Setting Unit

[0136] 240 Identification Processing Unit

[0137] 260 Auxiliary Control Unit

[0138] 280 storage devices

[0139] 290 communication device

[0140] 700 range

[0141] 2000 Computer

[0142] 2010 Main Controller

[0143] 2012CPU

[0144] 2014 RAM

[0145] 2020 Input / Output Controller

[0146] 2022 Communication Interface

[0147] 2024 Flash

[0148] 2026ROM

[0149] 2040 input / output chips.

Claims

1. An auxiliary control device, characterized in that: have: a first acquisition unit that acquires position information of a first moving object detected from an image captured by the imaging device; a second acquisition unit that acquires position information of the second moving object measured at each of the second moving objects; an identification processing unit that performs identification processing in which the position information of the first moving object and the position information of the second moving object are compared for each combination of the first moving object and the second moving object, and a combination of moving objects whose position information has a consistency higher than a predetermined value is identified as one moving object; as well as an assistance control section that performs control for traffic assistance for a mobile object based on the position information of the mobile object obtained by applying the recognition processing to the position information of the first mobile object and the position information of the second mobile object, The recognition processing unit limits the position information to be processed by the recognition process, out of the position information of the first moving object and the position information of the second moving object, to position information within a predetermined range set based on the position of the imaging device.

2. The auxiliary control device according to claim 1, wherein: The predetermined range is a predetermined fixed range including a position where the imaging device is installed.

3. The auxiliary control device according to claim 1, wherein: It also has a range setting unit, which selects at least four position information having outermost coordinate components for two-axis coordinate axes determined within the predetermined range from the position information of the first moving object acquired by the first acquisition unit, and sets the range defined by a line passing through each of the selected position information as the predetermined range.

4. The auxiliary control device according to any one of claims 1 to 3, wherein: The recognition processing unit compares the position indicated by the position information of the first moving object and the position indicated by the position information of the second moving object for each combination of the first moving object and the second moving object, and recognizes a combination of mobile objects whose position consistency is higher than a predetermined value as one moving object.

5. The auxiliary control device according to claim 4, wherein: The recognition processing unit recognizes a combination of moving objects whose degree of coincidence of the positions is higher than a predetermined value within a predetermined time as one moving object.

6. The auxiliary control device according to claim 4, wherein: The recognition processing unit will also compare the moving direction determined based on the history of the position information of the first mobile object and the moving direction determined based on the history of the position information of the second mobile object for each combination of the first mobile object and the second mobile object, and identify the combination of mobile objects whose position consistency is higher than a predetermined value and whose moving direction consistency is higher than a predetermined value as one mobile object.

7. The auxiliary control device according to claim 6, wherein: The recognition processing unit recognizes a combination of moving objects whose positions have a consistency higher than a predetermined value within a predetermined time and whose moving directions have a consistency higher than a predetermined value within a predetermined time as one moving object.

8. The auxiliary control device according to claim 4, wherein: The recognition processing unit performs the following processing for each combination of the first moving object and the second moving object: determining over time whether the consistency of the positions is above a predetermined value, Whenever it is determined that the degree of coincidence at the position is higher than a predetermined value, the evaluation value is increased by a predetermined value, and whenever it is determined that the degree of coincidence at the position is lower than a predetermined value, the evaluation value is decreased by a predetermined value, The recognition processing unit performs recognition processing for recognizing a combination of moving objects whose evaluation values ​​reach a predetermined value as one moving object.

9. The auxiliary control device according to claim 6, wherein: The recognition processing unit performs the following processing for each combination of the first moving object and the second moving object: determining over time whether the consistency of the position is higher than a predetermined value and the consistency of the moving direction is higher than a predetermined value, Whenever it is determined that the degree of consistency of the position is higher than a predetermined value and the degree of consistency of the moving direction is higher than a predetermined value, the evaluation value is increased by a predetermined value, and whenever it is determined that the degree of consistency of the position is lower than a predetermined value or the degree of consistency of the moving direction is lower than a predetermined value, the evaluation value is decreased by a predetermined value, Recognition processing is performed to recognize a combination of moving objects whose evaluation values ​​reach a predetermined value as one moving object.

10. The auxiliary control device according to any one of claims 1 to 3, wherein: When it is determined that there are mobile objects approaching each other within a predetermined time based on the position information of the mobile objects obtained by applying the recognition processing to the position information of the first mobile object and the position information of the second mobile object, the auxiliary control unit performs control for sending an alarm to at least one of the mobile objects that may approach each other.

11. The auxiliary control device according to claim 10, wherein: the assist control section, based on a history of position information of the mobile object obtained by applying the recognition process to the position information of the first mobile object and the position information of the second mobile object, predicting the position of a moving object that moves from a shooting range of the shooting device to outside the shooting range, Based on the predicted position of the moving object and the position indicated by the position information of other moving objects, it is determined whether there are moving objects approaching each other within a predetermined time.

12. An auxiliary control method, characterized in that: have: a step of acquiring position information of a first moving object detected from an image captured by a capturing device; A step of acquiring position information of the second moving object measured in each of the second moving objects; a step of performing recognition processing, in which the position information of the first moving object and the position information of the second moving object are compared for each combination of the first moving object and the second moving object, and a combination of mobile objects whose position information has a consistency higher than a predetermined value is recognized as one mobile object; as well as performing control for traffic assistance for a mobile object based on the position information of the mobile object obtained by applying the recognition process to the position information of the first mobile object and the position information of the second mobile object, In the step of performing the recognition processing, the position information of the first moving object and the position information of the second moving object, which is the object of the recognition processing, is limited to position information within a predetermined range set based on the position of the imaging device.

13. A computer-readable storage medium, characterized in that: A program for causing a computer to function as the auxiliary control device according to any one of claims 1 to 11 is stored.

Images