Multi-sensor fusion positioning method based on GIS and computer vision

Abstract

The invention provides a multi-sensor fusion positioning method based on GIS and computer vision, and the method comprises the steps: S1, obtaining the GPS data of a vehicle, and obtaining a road network deviation value according to the comparison between the GPS data and wharf road network data; s2, judging whether the road network deviation value is greater than a positioning error threshold value or not: if so, judging that a signal source of the GPS data is a red GIS and turning to the step S3; if not, judging that the signal source of the GPS data is a yellow GIS and turning to the step S4; s3, acquiring license plate information of a vehicle, performing vehicle license plate recognition and screening based on digital camera videos output by each camera device to obtain a target video, performing back calculation according to the target video on the three-dimensional map to obtain a GIS position, and performing positioning compensation on the GPS data to obtain compensated GPS data; step S4, judging whether the vehicle is located in a pre-defined video fusion area or not: if so, turning to step S3; and if not, outputting the GPS data. The method has the beneficial effects that accurate positioning of the vehicle can be realized, and the accuracy of GPS data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sensor positioning, and in particular to a multi-sensor fusion positioning method based on GIS and computer vision. Background Art

[0002] In the operation process of the port container terminal, the gantry crane is the standard container loading and unloading equipment on the dock. The multi-layer steel frame structure of the gantry crane will cause obvious multipath effect to the global satellite positioning system receiver under the gantry crane, resulting in low satellite positioning accuracy or even inability to complete position calculation. When a large container ship docks, there are usually multiple gantry cranes operating side by side, which will form a steel tunnel on the dock surface, causing the satellite positioning system to completely fail, resulting in the positioning technology based on high-precision satellite positioning being unable to work on the dock surface.

[0003] An existing vehicle positioning method is a high-precision satellite navigation system positioning technology based on RTK (Real-time kinematic carrier phase differential technology). It obtains the centimeter-level position information of the vehicle itself by receiving positioning data and differential signal data from satellite systems such as GPS, GLONASS and Beidou. However, the problem with this method is that the vehicle must be in a very open place to calculate its own precise position information, and the operating scenes of port terminals are blocked by many bridge cranes, so accurate positioning cannot be achieved through this method.

[0004] At present, the environment of shipping container terminals is complex and easily affected by interference from signals, metal, etc. A single GPS geographic location information cannot accurately locate the equipment in real time. Data correction, data fusion, and positioning compensation for external container trucks and other operating equipment have become issues that need to be addressed urgently. The original technology for positioning container trucks and other equipment only relies on GPS, but due to the complex environment, GPS may not be accurate in some situations. The existing technology relies purely on GPS. Secondly, relying solely on GPS for operational safety inspections is inconvenient, vehicle data and images are not unified, the degree of abstraction is too high, and the efficiency of troubleshooting and safety inspections is low. Summary of the invention

[0005] The technical problem to be solved by the present invention is to achieve accurate positioning of the vehicle and improve the accuracy of GPS data. In order to overcome the defects of the above-mentioned prior art (or related technology), the present invention provides a multi-sensor fusion positioning method based on GIS and computer vision.

[0006] The present invention provides a multi-sensor fusion positioning method based on GIS and computer vision. Aiming at the dock yard scene in advance, a plurality of camera devices are set in the vehicle entry and exit area in the dock yard scene. The dock vehicle positioning method comprises the following steps: Step S1, obtaining GPS data of vehicles entering the terminal yard scene, and obtaining a road network deviation value according to the comparison between the GPS data and the preset terminal road network data; Step S2, determining whether the road network deviation value is greater than a preset positioning error threshold: If yes, it is determined that the signal source of the GPS data is red GIS and the process goes to step S3; If not, it is determined that the signal source of the GPS data is yellow GIS and the process goes to step S4; Step S3, obtaining the license plate information of the vehicle, performing vehicle license plate recognition based on the digital camera video output by each of the camera devices, screening and obtaining a target video matching the license plate information, performing positioning compensation on the GPS data based on the GIS position obtained by reverse calculation of the target video on the three-dimensional map, obtaining compensated GPS data and outputting the compensated GPS data; Step S4, judging whether the vehicle is located in a predefined video fusion area based on the GPS data: If yes, go to step S3; If not, the GPS data is output.

[0007] Compared with the prior art, the multi-sensor fusion positioning method based on GIS and computer vision in this application has the following advantages: In the present application, step S1 is used to acquire GPS data and calculate the road network deviation value, step S2 is used to determine the red GIS and the yellow GIS, step S3 is used to acquire license plate information and perform positioning compensation on GPS data, step S4 is used to determine the video fusion area, and the license plate information is used as the core element to screen the digital camera video to obtain the matching target video, and then the GIS position is obtained by inverse calculation to perform positioning compensation on the GPS data, and the accuracy of the GPS data is corrected by the video elements, so as to achieve accurate positioning of the vehicle and improve the accuracy of the GPS data.

[0008] In a possible implementation, in step S3, the license plate information and the vehicle pixel position are obtained by sensing the target video recognition, the video pixel coordinates of the vehicle pixel position are inversely calculated to obtain the three-dimensional spatial position, and then the GPS data of the vehicle is associated with the license plate information to correct it to obtain the compensated GPS data.

[0009] In a possible implementation, step S3 includes: Step S31, obtaining the license plate information of the vehicle, and performing vehicle license plate recognition based on the digital camera video output by each of the camera devices; Step S32, determine whether the license plate can be recognized: If yes, go to step S33; If not, then transfer to step S34; Step S33, based on the identified license plate, the target video matching the license plate information is screened out, the GIS position is obtained by reverse calculation according to the target video on the three-dimensional map, positioning compensation is performed on the GPS data to obtain the compensated GPS data and output; Step S34, obtaining and obtaining a GPS compensation weight for distinguishing the video fusion area according to the sensing device pixels and the flood range; Step S35, performing positioning compensation on the GPS according to the GPS compensation weight to obtain the compensated GPS data and output it.

[0010] In a possible implementation, step S5 includes: Step S51, determining whether the GPS compensation weight is greater than a preset threshold: If yes, output the GPS data; If not, go to step S52; Step S52, compensating the GPS by matching the GPS parameters of the vehicle's nearest target with the same signal source according to a proximity algorithm to obtain the compensated GPS data and output it.

[0011] In a possible implementation manner, the video fusion area set in step S4 represents an area not covered by the camera device.

[0012] In a possible implementation, a license plate recognition camera is provided in the vehicle entry and exit area in the terminal yard scene, and the license plate information of the vehicle is acquired through the license plate recognition camera in step S3.

[0013] In a possible implementation, the vehicle includes an internal container truck and an external container truck, the internal container truck uses a port-specific license plate, and the external container truck uses a social vehicle license plate. In step S3, the port-specific license plate or the social vehicle license plate is obtained as the license plate information of the vehicle.

[0014] In a possible implementation manner, in step S3, after obtaining the license plate information of the vehicle, the step further includes: The license plate information is stored in a data server for retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the steps of the present invention; Figure 2 is a specific flow chart of step S3 of the present invention; Figure 3is a specific flow chart of step S5 of the present invention; Figure 4 It is a schematic diagram of camera installation in the empty container yard of the present invention; Figure 5 A schematic diagram of camera installation at the front of the wharf of the present invention; Figure 6 It is a schematic diagram of camera installation in the yellow GIS area of ​​the present invention. DETAILED DESCRIPTION

[0016] First, those skilled in the art should understand that these implementations are only used to explain the technical principles of the embodiments of the present application, and are not intended to limit the protection scope of the embodiments of the present application. Those skilled in the art can make adjustments to them as needed to adapt to specific application scenarios.

[0017] The present application is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] See also Figure 1 The present application embodiment discloses a multi-sensor fusion positioning method based on GIS and computer vision, including: Step S1, obtaining GPS data of vehicles entering the terminal yard scene, and obtaining a road network deviation value by comparing the GPS data with pre-set terminal road network data; Step S2: determine whether the road network deviation value is greater than a preset positioning error threshold: If yes, it is determined that the signal source of the GPS data is red GIS and the process goes to step S3; If not, it is determined that the signal source of the GPS data is yellow GIS and the process goes to step S4; Step S3, obtaining the vehicle license plate information, performing vehicle license plate recognition based on the digital camera video output by each camera device, screening out the target video matching the license plate information, performing positioning compensation on the GPS data based on the GIS position obtained by reverse calculation according to the target video on the three-dimensional map, obtaining the compensated GPS data and outputting it; Step S4, judging whether the vehicle is located in a pre-defined video fusion area based on GPS data: If yes, go to step S3; If not, the GPS data is output.

[0019] Continue to see Figure 1This application adopts computer vision technology to solve the problem of inaccurate original RTK measurement accuracy caused by complex environmental interference. Taking the container terminal yard scene as an example, it aims to solve the perception and positioning problem of the automated operation of the electric control equipment in the yard in the complex dynamic environment of the yard. It proposes a method based on the original RTK data information, through the position calibration of the yard camera device, the transformation of the spatial coordinate system, and the transformation of the equipment target from the map to the monitoring screen, and from the monitoring screen to the map; by converting the high-altitude drone image with the actual screen coordinates, through the fusion of multiple sensors such as GPS positioning sensors, wireless communication sensors, optical recognition sensors, RFID sensors, and vehicle load sensors, with GPS position as the link, the multi-level and multi-granular equipment position information is unified. Experimental tests show that the improved fused location information has greatly improved the accuracy of the original GIS.

[0020] Continue to see Figure 1 In this application, the multi-sensor fusion method is mainly used to coordinate and correct the camera device and the GPS positioning sensor. In the face of the current problem that the positioning is not real-time and accurate enough, a multi-dimensional perception application system based on GIS is built, and the technical means of three-dimensional video fusion GIS positioning + GPS positioning are used to compensate the positioning information, that is, GPS data, to achieve the effect of GIS virtual-real fusion application. Specifically, it mainly solves the positioning correction, verification, fusion and compensation of the yellow GIS area and the red GIS area. The technical implementation plan is as follows: 1. Yellow GIS, red GIS judgment Set the terminal road network data and GIS positioning error threshold of the terminal yard scene, compare the input GPS data with the terminal road network data to calculate the road network error value, and then verify it with the GIS positioning error threshold to determine whether the signal source is red GIS or yellow GIS; 2. Red GIS positioning compensation Red GIS requires accurate vehicle video position for positioning compensation, so in practical applications it must be able to identify the license plate information of the truck; Based on the red GIS truck license plate recognition, the GIS position is calculated by the digital camera video on the three-dimensional map to compensate the GPS data with positioning errors, that is, the license plate and vehicle pixel position are recognized by the perception device video, the video pixel coordinates are calculated to get the three-dimensional space position, and then the GPS data of the vehicle is associated with the license plate information to correct the vehicle's GPS data; 3. Yellow GIS positioning compensation Yellow GIS requires the perception device to be able to identify the vehicle in the video image, and will be divided into the following GIS positioning compensation scenarios: (1) Video fusion area where the license plate can be identified The GIS position is calculated by the digital camera video on the three-dimensional map to compensate for the GPS data with positioning errors. That is, the license plate and vehicle pixel position are recognized by the video of the sensing device, and the three-dimensional space position is calculated by the video pixel coordinates. Then, the GPS data of the vehicle is associated with the license plate to correct the vehicle's GPS data. (2) The license plate cannot be recognized, but the video fusion area of ​​the vehicle in the video image can be recognized Video fusion position calculation: The device identifies the vehicle pixel position and reversely calculates the three-dimensional space position from the video pixel coordinates; Weight calculation: The GPS compensation weight of the video fusion area is distinguished according to the perception device pixels, flood range, etc. When the weight is less than the preset threshold, the input GPS data is combined with the fusion position calculation, and the nearest target with the same GPS signal source is matched according to the proximity algorithm to give GPS signal compensation; If the weight is greater than the preset threshold: return the original GPS data; And considering that in some areas, the GPS data recognized by the GPS system and the GIS data calculated by the video may be inaccurate, resulting in the inability to meet the positioning compensation requirements, in this scenario, a camera will be installed to recognize the vehicle license plate; (3) Non-video fusion area not covered by the video camera The GPS data reported by the original GPS system shall prevail.

[0021] See also Figure 2 , step S3 comprises: Step S31, obtaining the vehicle license plate information, and performing vehicle license plate recognition based on the digital camera video output by each camera device; Step S32, determine whether the license plate can be recognized: If yes, go to step S33; If not, then transfer to step S34; Step S33, based on the identified license plate, a target video matching the license plate information is screened out, and the GIS position is obtained by reverse calculation according to the target video on the three-dimensional map, and the GPS data is positioned and compensated to obtain the compensated GPS data and output; Step S34, obtaining and obtaining GPS compensation weights for distinguishing video fusion areas according to the sensing device pixels and the flood range; Step S35, performing positioning compensation on the GPS according to the GPS compensation weight to obtain compensated GPS data and output it.

[0022] See also Figure 3 , step S5 comprises: Step S51, determine whether the GPS compensation weight is greater than a preset threshold: If yes, then output GPS data; If not, go to step S52; Step S52, compensating the GPS by matching the GPS parameters of the nearest target vehicle with the same signal source according to the proximity algorithm to obtain the compensated GPS data and output it.

[0023] Continue to see Figure 1 Compared with the existing equipment positioning solutions, the method of this application has a significant improvement in the positioning of external container trucks, greatly reducing safety hazards, and providing important support for the management and control of external container trucks; it provides key technical support for port safety production, emergency management, strategic command, and directive action; in terms of port safety production, emergency management, and strategic command, it is the first to propose a terminal vehicle positioning method based on GIS and computer vision. Taking the container terminal yard as an example, the original RTK data information of the equipment on the yard is integrated with multiple sensors such as the ground camera perspective and the aerial drone perspective. It is an innovative data fusion technology that brings together all-factor data in two or three dimensions, above and below the ground, static and dynamic, historical and present, and dynamically links things with location as the link, with multiple angles, multiple levels, and multiple granularities.

[0024] Continue to see Figure 1 In this application, different video fusion compensation and correction methods are used for vehicle positions due to different degrees of GIS data offset. Different camera types and positions need to be configured for red GIS and yellow GIS. Red GIS requires accurate vehicle video position compensation and needs to identify license plate information, while yellow GIS only needs to identify vehicles. Internal container trucks use port-specific license plates, which are different from social vehicle license plates. Universal license plate recognition algorithms cannot recognize internal container truck license plates. It is necessary to train the license plate recognition algorithm to support both social license plates and port-specific license plates. In order to improve the accuracy of license plate recognition in the port, internal container truck license plates need to be unified. The following examples illustrate the camera position settings for the red GIS area and the yellow GIS area: (1) Red GIS The red GIS area is temporarily analyzed according to the empty container yard and the front of the wharf where the positioning is inaccurate. Considering the restrictions of the quay crane and yard crane operation poles at the yard and the front of the wharf, it is recommended to install license plate recognition cameras on the port machinery. Figure 4 For empty container yards, it is not possible to erect poles in empty container yards. It is recommended that the camera be installed on the quay crane legs. The camera needs to support universal license plate and port license plate recognition. The installation height is about 6m and the angle supports 2 lanes. See Figure 5 For the front of the wharf, it is recommended to install the camera on the outrigger and the lower beam of the quay crane. The outrigger camera monitors the road outside the quay crane, and the camera on the beam monitors the road below the quay crane. A single camera supports 2 roads. (2) Yellow GIS See also Figure 6The yellow GIS area is used for vehicle identification algorithm. The existing port road monitoring equipment can be used in combination with the back-end vehicle intelligent identification algorithm for vehicle identification. If there is no video coverage, it is recommended to install it on the port lighthouse or lamp pole. The newly installed video equipment can be combined with the back-end vehicle intelligent identification algorithm for vehicle identification.

[0025] In the description of the present application, the description with reference to the terms "one embodiment", "some embodiments", "in the present embodiment", "specific example", or "some examples" etc. means that the specific features, mechanisms, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, mechanisms, materials or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0026] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A terminal vehicle positioning method based on GIS and computer vision, characterized in that: Aiming at the dock yard scene in advance, multiple camera devices are set in the vehicle entry and exit area in the dock yard scene. The dock vehicle positioning method includes the following steps: Step S1, obtaining GPS data of vehicles entering the terminal yard scene, and obtaining a road network deviation value according to the comparison between the GPS data and the preset terminal road network data; Step S2, determining whether the road network deviation value is greater than a preset positioning error threshold: If yes, it is determined that the signal source of the GPS data is red GIS and the process goes to step S3; If not, it is determined that the signal source of the GPS data is yellow GIS and the process goes to step S4; Step S3, obtaining the license plate information of the vehicle, performing vehicle license plate recognition based on the digital camera video output by each of the camera devices, screening and obtaining a target video matching the license plate information, performing positioning compensation on the GPS data based on the GIS position obtained by reverse calculation of the target video on the three-dimensional map, obtaining compensated GPS data and outputting the compensated GPS data; Step S4, judging whether the vehicle is located in a predefined video fusion area based on the GPS data: If yes, go to step S3; If not, the GPS data is output.

2. The method for locating a terminal vehicle according to claim 1, characterized in that: In step S3, the license plate information and the vehicle pixel position are obtained by sensing the target video recognition, the video pixel coordinates of the vehicle pixel position are inversely calculated to obtain the three-dimensional space position, and then the GPS data of the vehicle is associated with the license plate information to correct it to obtain the compensated GPS data.

3. The method for locating a terminal vehicle according to claim 1, characterized in that: The step S3 comprises: Step S31, obtaining the license plate information of the vehicle, and performing vehicle license plate recognition based on the digital camera video output by each of the camera devices; Step S32, determine whether the license plate can be recognized: If yes, go to step S33; If not, then transfer to step S34; Step S33, based on the identified license plate, the target video matching the license plate information is screened out, the GIS position is obtained by reverse calculation according to the target video on the three-dimensional map, positioning compensation is performed on the GPS data to obtain the compensated GPS data and output; Step S34, obtaining and obtaining a GPS compensation weight for distinguishing the video fusion area according to the sensing device pixels and the flood range; Step S35, performing positioning compensation on the GPS according to the GPS compensation weight to obtain the compensated GPS data and output it.

4. The method for locating a terminal vehicle according to claim 3, characterized in that: The step S5 comprises: Step S51, determining whether the GPS compensation weight is greater than a preset threshold: If yes, output the GPS data; If not, go to step S52; Step S52, compensating the GPS by matching the GPS parameters of the vehicle's nearest target with the same signal source according to a proximity algorithm to obtain the compensated GPS data and output it.

5. The method for locating a terminal vehicle according to claim 1, characterized in that: The video fusion area set in step S4 represents an area not covered by the camera device.

6. The method for locating a terminal vehicle according to claim 1, characterized in that: The vehicle entry and exit area in the terminal yard scene is provided with a license plate recognition camera, and the license plate information of the vehicle is obtained through the license plate recognition camera in step S3.

7. The method for locating a terminal vehicle according to claim 1, characterized in that: The vehicles include internal container trucks and external container trucks. The internal container trucks use port-specific license plates, and the external container trucks use social vehicle license plates. In step S3, the port-specific license plates or the social vehicle license plates are obtained as the license plate information of the vehicles.

8. The method for locating a terminal vehicle according to claim 1, characterized in that: In the step S3, after obtaining the license plate information of the vehicle, the step further includes: The license plate information is stored in a data server for retrieval.

Images