Underground garage vehicle searching system and method based on machine vision

Abstract

The invention provides an underground garage vehicle searching system and method based on machine vision. The system comprises an execution module and a detection module. The software system comprises an image processing module, an information storage module and a man-machine interaction module. The execution module carries a power supply unit, a driving unit, a control unit and the like. The detection module comprises an industrial personal computer and a depth camera and is used for detecting vehicles and license plate numbers on the parking spaces. An image processing module of the software part is based on image information transmitted by a depth camera, an improved neural network is used for completing recognition of vehicles and license plate numbers, and an information storage module is used for storing information about whether vehicles exist in all parking spaces or not and information about the license plate numbers. The man-machine interaction module interacts with a user through a WeChat applet. The method comprises the following steps of: detecting whether a vehicle exists in each parking space and information of a license plate number by using the image detection device, and storing data; and when the idle parking space is found, the idle parking space is retrieved and the user is guided to the designated position. According to the invention, convenience is provided for vehicle position finding.

Description

Technical Field

[0001] The present application relates to the field of positioning and navigation technology, and in particular to a system and method for finding a vehicle in an underground garage based on machine vision. Background Art

[0002] With the advancement of science and technology and the acceleration of urbanization, there are more and more garages in large commercial and residential areas. The rapid growth of the number of private cars has often made people face the problem of difficulty in parking and finding cars. The existing parking space identification of garages often manages parking spaces by taking IC cards or scanning license plates at the entrances and exits, that is, the owner manages the parking space by swiping the card. The existing system can only play the role of charging entrances and exits, and cannot indicate the specific location of the vacant parking spaces, which is not conducive to management. At the same time, the existing garage requires the owner to find an empty parking space and memorize the parking position of the vehicle after entering the garage. For facilities such as shopping malls, airports, and railway stations equipped with large parking lots, it is extremely inconvenient for the owner to find the vehicle, and it will also make the management of the garage more chaotic. How to better facilitate the owner to find the target vehicle or find a suitable parking space and manage it based on relevant information has become a hot issue. There is no better solution for the existing underground garage navigation. Some indoor positioning navigation methods are difficult to directly apply to garage navigation. The existing autonomous driving vehicles can complete the search for vacant parking spaces, but they do not help the management of parking lots and are not universal for most vehicles. Summary of the invention

[0003] The present application provides a system and method for finding a car in an underground garage based on machine vision, which can be used to solve the technical problem that parking space management cannot perform navigation.

[0004] The present application provides an underground garage car search system based on machine vision, the system comprising hardware parts such as an execution module, a detection module, an image processing module, an information storage module and a human-computer interaction module;

[0005] The execution module comprises a garage execution module, on which a power supply unit (1), a drive unit (2), and a control unit (3) are mounted;

[0006] The detection module comprises an industrial computer (4) and a depth camera D455 (5), which are used to collect image information of the vehicle; the execution module is directly connected to the detection module to drive the normal operation of the detection module;

[0007] The image processing module uses the improved YOLOv8 combined with the CBResNet50-DCN algorithm to recognize vehicles and license plates based on the image information transmitted by the depth camera, and uses the multi-scale fusion technology BiFPN and the accelerated inference network TensorRT to improve the accuracy and speed of detection;

[0008] The information storage module stores information based on the Onenet cloud platform and the MySQL database;

[0009] The human-computer interaction module is a WeChat applet that interacts with users.

[0010] Furthermore, the power supply unit (1) provides a 24V power supply to the drive unit (2) and the control unit (3);

[0011] The microprocessor of the control unit (3) is used to issue control instructions. Under the control of the control unit (3), the drive unit (2) drives the chassis steering wheel to complete the work of steering and adjusting the rotation speed, and finally completes the control of the movement of the execution module.

[0012] Furthermore, the detection module detects the image information during the motion process, uses the YOLOv8 model in the image processing module to identify the target license plate area, and then uses CBResNet50-DCN to identify the character information in the corresponding area one by one, and finally splices the complete license plate information; among them, the YOLOv8 model integrates the BiFPN network and the TensorRT network;

[0013] Among them, the specific way of integrating BiFPN is as follows: the first connection layer of the head connects the upsampled features with the P4 layer in the backbone network, and the module used for the connection is changed from the Concat module to the BiFPN_Concat2 module, and the parameters remain unchanged; the second connection layer of the head connects the upsampled features with the third feature layer in the backbone network, and the module used for the connection is changed from the Concat module to the BiFPN_Concat2 module, and the parameters remain unchanged; the third connection layer of the head connects the upsampled features with the fourth feature layer in the head network, and the module used for the connection is changed from the Concat module to the BiFPN_Concat3 module, and the number of channels of the input feature map is changed to [-1, 6, 12]; the fourth connection layer of the head connects the upsampled features with the fifth feature layer in the head network, and the module used for the connection is changed from the Concat module to the BiFPN_Concat2 module, and the parameters remain unchanged;

[0014] The way to integrate TensorRT is to use the tensorRT acceleration network to modify the model file after obtaining the trained model file, quantize the FP16 precision model of the original model, and convert the floating-point parameters into fixed-point parameters; merge multiple convolutional layers and pooling layers into one layer to reduce access time and memory time.

[0015] Furthermore, the information storage module receives the detection results of the image processing module, stores the data information through WiFi and MQTT protocol with the help of Onenet cloud platform, and links to MySQL database; wherein the data information includes whether there is a car in each parking space and the corresponding license plate information; Boolean quantities 0 and 1 are used to indicate whether there is a car, 0 indicates no car, and 1 indicates a car; the corresponding license plate information is represented in string form; when the human-computer interaction module provides information, the information storage module will search in the stored database, and after finding the appropriate parking space information, return it to the execution module for control.

[0016] Furthermore, the human-computer interaction module is mainly a WeChat applet. Users put forward service requirements through the WeChat applet, and the applet returns the demand information to the information storage module for subsequent operations.

[0017] The present application also provides a method for finding a car in a garage based on machine vision. The method provided by the present application is implemented using the system provided by the present application. The method includes the following steps:

[0018] Step 1: When the user does not issue any instructions, the execution module checks each parking space one by one according to the built map combined with the detection module to see if there is a car. If there is a car, the license plate number data information is identified and uploaded to the information storage module. When all inspections are completed, the information storage module will save the information of whether there is a car in each parking space and its license plate number;

[0019] Step 2: When the user drives into the garage, the user issues a command to find an empty parking space through the human-computer interaction module. After receiving the command, the execution module drives to the user's location. The information storage module retrieves the empty parking spaces based on the stored data information and returns the empty parking spaces and the empty positions to the execution module. The execution module guides the user to a suitable parking space.

[0020] Step 3: The detection module records the information of the license plate number and uploads it to the information storage module, modifies the Boolean value of the parking space information from 0 to 1, and stores the license plate number in the position of the license plate information in the form of a string;

[0021] Step 4. When the user plans to drive away and needs to find his car in the garage, he sends a car search command through the human-computer interaction module and enters his license plate number. The information storage module retrieves the target parking space through the license plate number, and lets the execution module lead the user to find the parking space. The license plate number on the parking space is checked with the target license plate number through the detection module. After confirmation, it means that the user has found the car. At the same time, the information storage module clears the license plate information of the parking space and changes the parking space information from 1 to 0, indicating that there is no car.

[0022] Furthermore, step 1 of license plate recognition is implemented by the following method:

[0023] Step 1.1, Preparatory stage: Integrate the YOLOv8 network into the BiFPN network to modify the fully connected layer, use the modified YOLOv8 network to train the pre-prepared image dataset of cars and license plates to obtain the trained pt model file, and use the TensorRT network to improve the obtained pt file to obtain the engine file, which is the model file used for the final recognition.

[0024] Step 1.2, parse the input image information according to the model file to get the output result and get the bounding box coordinates;

[0025] Step 1.3: Preprocess the grayscale and binarization according to the detected license plate coordinate information;

[0026] Step 1.4, continue to use the CBResNet50-DCN network to recognize each character on the license plate;

[0027] Step 1.5: Identify and assemble the complete license plate number information based on the character output results and save it in the system.

[0028] Furthermore, the method for license plate detection in step 3 is consistent with the method in step 1.

[0029] Furthermore, the improved YOLOv8 model described in step 1.1 integrates the BiFPN network and the TensorRT network on the basis of the original YOLOv8 model. The BiFPN network improves the model performance and detection accuracy by building bidirectional links and weighted feature integration of feature pyramids, and the TensorRT network improves the detection speed by optimizing the model and reducing redundant calculations.

[0030] Compared with the prior art, the present invention has the following significant advantages: (1) the license plate number, which is a key element for identifying automobile information, is used to effectively combine machine vision with Internet of Things technology; (2) the image processing module uses the multi-scale fusion technology BiFPN and the accelerated inference network TensorRT to improve the accuracy and speed of detection; and (3) combined with the matching garage execution module, it helps users find their cars. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is an execution structure model diagram provided by an embodiment of the present application;

[0032] Figure 2 is a schematic diagram of a detection module provided in an embodiment of the present application;

[0033] Figure 3 is an algorithm flow chart of the image detection module provided in the embodiment of the present application;

[0034] Figure 4 is an interface diagram of a human-computer interaction module provided in an embodiment of the present application;

[0035] Figure 5 It is a flow chart of the human-computer communication algorithm provided in the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0037] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.

[0038] See also Figures 1 to 5 , Figure 1 It is the execution module model diagram; Figure 2 is a schematic diagram of the detection module; Figure 3 It is the algorithm flow chart of the image detection module; Figure 4 It is the interface diagram of the human-computer interaction module; Figure 5 It is a flow chart of the human-computer communication algorithm.

[0039] like Figure 1 As shown, the execution module of the hardware system is mainly a garage execution module, which is equipped with a power supply unit (1), a drive unit (2), a control unit (3), etc. The motion control program of the execution module is encapsulated in the control unit (3) carried on it.

[0040] like Figure 2 As shown, the depth camera used in the detection module of the hardware system is Intel D455, which has the advantages of powerful depth perception, multi-mode operation, strong adaptability to low-light environments, and easy integration. The depth camera is combined with the YOLOv8 algorithm for visual detection, which can very efficiently and accurately complete the function of visual image detection. The depth camera is connected to the industrial computer (4) via a data cable to complete the image solution.

[0041] like Figure 3 The algorithm flow chart of the image detection module of the software system is shown in FIG. The algorithm uses the improved YOLOv8 algorithm based on the original image information to recognize the approximate area of ​​the license plate and parse the frame coordinates, grayscales and binarizes the image, and then uses the CBResNet50-DCN network to recognize each character one by one, and then splices the complete license plate number information for final output.

[0042] like Figure 4 As shown in the figure, it is the main interface design diagram of the WeChat applet of the human-computer interaction module. Figure 4(1) There are two function buttons to choose from, namely "Drive into the parking lot" and "Take you to find your car". Clicking "Drive into the parking lot" prompts the system to help the user find the target vacant parking space. Clicking "Take you to find your car" jumps the interface to Figure 4 (2) Enter the license plate number and the system will help the user find the parking space where the target car is located.

[0043] like Figure 4 As shown, it is the overall algorithm flow chart. When the user makes an application, the robot is controlled to lead the user to the required location according to the processing results of the information storage module, and the verification is performed according to the detection results of the detection module. After completing the task, the information of the information storage module is updated.

[0044] like Figure 5 As shown, this is an algorithm flow chart of the software system for implementing human-computer interactive communication.

[0045] The execution module will conduct inspections in the garage in advance and save the data to the information storage module based on the inspection results. The specific data mainly includes whether there is a car in each garage and the license plate number of the parking space. When the user drives into the garage, the user can issue a command to find an empty parking space through the human-computer interaction module "Drive into the garage". After receiving the command, the execution module will rush to the user and lead the user to the empty parking space based on the empty parking space found in the information storage module, and then end the task. The detection module will record the information of the license plate number and upload it to the information storage module, change the Boolean value of the parking space information from 0 to 1, that is, there is a car, and store the license plate number in the position of the license plate information in the form of a string.

[0046] When the user plans to drive away, he can issue a command of "take you to find the car" through the human-computer interaction module and enter his license plate number. The information storage module retrieves the target parking space through the license plate number, and lets the execution module lead the user to find the parking space. The license plate number on the parking space is checked with the target license plate number through the detection module. After confirmation, it means that the user has found the car. The information storage module clears the license plate information of the parking space and changes the parking space information from 1 to 0, indicating that there is no car.

[0047] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

[0048] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A machine vision-based underground garage car search system, characterized in that: The system includes hardware parts such as an execution module, a detection module, an image processing module, an information storage module and a human-computer interaction module; The execution module comprises a garage execution module, and the garage execution module is equipped with a power supply unit (1), a drive unit (2), and a control unit (3); The detection module comprises an industrial computer (4) and a depth camera (5) for collecting image information of the vehicle; the execution module is directly connected to the detection module to drive the normal operation of the detection module; The image processing module uses the improved YOLOv8 combined with the CBResNet50-DCN algorithm to recognize vehicles and license plates based on the image information transmitted by the depth camera, and adopts the multi-scale fusion technology BiFPN and the accelerated inference network TensorRT; The information storage module stores information based on the Onenet cloud platform and the MySQL database; The human-computer interaction module is a WeChat applet that interacts with users.

2. The machine vision-based garage car search system according to claim 1, characterized in that: The power supply unit (1) provides a 24V power supply to the drive unit (2) and the control unit (3); The microprocessor of the control unit (3) is used to issue control instructions. Under the control of the control unit (3), the drive unit (2) drives the chassis steering wheel to complete the work of steering and adjusting the rotation speed, and finally completes the control of the movement of the execution module.

3. The machine vision-based garage car search system according to claim 1, characterized in that: The detection module detects the image information during the motion process, uses the YOLOv8 model in the image processing module to identify the target license plate area, and then uses CBResNet50-DCN to identify the character information in the corresponding area one by one, and finally splices the complete license plate information; among them, the YOLOv8 model integrates the BiFPN network and the TensorRT network; Among them, the specific way of integrating BiFPN is as follows: the first connection layer of the head connects the upsampled features with the P4 layer in the backbone network, and the module used for the connection is changed from the Concat module to the BiFPN_Concat2 module, and the parameters remain unchanged; the second connection layer of the head connects the upsampled features with the third feature layer in the backbone network, and the module used for the connection is changed from the Concat module to the BiFPN_Concat2 module, and the parameters remain unchanged; the third connection layer of the head connects the upsampled features with the fourth feature layer in the head network, and the module used for the connection is changed from the Concat module to the BiFPN_Concat3 module, and the number of channels of the input feature map is changed to [-1, 6, 12]; the fourth connection layer of the head connects the upsampled features with the fifth feature layer in the head network, and the module used for the connection is changed from the Concat module to the BiFPN_Concat2 module, and the parameters remain unchanged; The way to integrate TensorRT is to use the tensorRT acceleration network to modify the model file after obtaining the trained model file, quantize the FP16 precision model of the original model, convert the floating-point parameters into fixed-point parameters; and fuse multiple convolutional layers and pooling layers into one layer.

4. The machine vision-based garage car search system according to claim 1, characterized in that: The information storage module receives the detection result of the image processing module, stores the data information through WiFi and MQTT protocol with the help of Onenet cloud platform, and links to MySQL database; the data information includes whether there is a car in each parking space and the corresponding license plate information; Boolean values ​​0 and 1 are used to indicate whether there is a car, 0 indicates no car, and 1 indicates a car; the corresponding license plate information is represented in string form; when the human-computer interaction module provides information, the information storage module will search in the stored database, and after finding the appropriate parking space information, return it to the execution module for control.

5. The machine vision-based garage car search system according to claim 1, characterized in that: The human-computer interaction module is mainly a WeChat applet. Users put forward service requirements through the WeChat applet, and the applet returns the demand information to the information storage module for subsequent operations.

6. A method for finding a car in a garage based on machine vision, the method being implemented using any system in claims 1 to 5, characterized in that: The method comprises the following steps: Step 1: When the user does not issue any instructions, the execution module checks each parking space one by one according to the built map combined with the detection module to see if there is a car. If there is a car, the license plate number data information is identified and uploaded to the information storage module. When all inspections are completed, the information storage module will save the information of whether there is a car in each parking space and its license plate number; Step 2: When the user drives into the garage, the user issues a command to find an empty parking space through the human-computer interaction module. After receiving the command, the execution module drives to the user's location. The information storage module retrieves the empty parking spaces based on the stored data information and returns the empty parking spaces and the empty positions to the execution module. The execution module guides the user to a suitable parking space. Step 3: The detection module records the information of the license plate number and uploads it to the information storage module, modifies the Boolean value of the parking space information from 0 to 1, and stores the license plate number in the position of the license plate information in the form of a string; Step 4. When the user plans to drive away and needs to find his car in the garage, he sends a car search command through the human-computer interaction module and enters his license plate number. The information storage module retrieves the target parking space through the license plate number, and lets the execution module lead the user to find the parking space. The license plate number on the parking space is checked with the target license plate number through the detection module. After confirmation, it means that the user has found the car. At the same time, the information storage module clears the license plate information of the parking space and changes the parking space information from 1 to 0, indicating that there is no car.

7. The method for finding a car in a garage based on machine vision according to claim 6, characterized in that: Step 1: License plate recognition is achieved by the following method: Step 1.1, Preparatory stage: Integrate the YOLOv8 network into the BiFPN network to modify the fully connected layer, use the modified YOLOv8 network to train the pre-prepared image dataset of cars and license plates to obtain the trained pt model file, and use the TensorRT network to improve the obtained pt file to obtain the engine file, which is the model file used for the final recognition. Step 1.2, parse the input image information according to the model file to get the output result and get the bounding box coordinates; Step 1.3: Preprocess the grayscale and binarization according to the detected license plate coordinate information; Step 1.4, continue to use the CBResNet50-DCN network to recognize each character on the license plate; Step 1.5: Identify and assemble the complete license plate number information based on the character output results and save it in the system.

8. The method for finding a car in a garage based on machine vision according to claim 6, characterized in that: The method for license plate detection in step 3 is the same as that in step 1.

9. The method for finding a car in a garage based on machine vision according to claim 7, characterized in that: The improved YOLOv8 model described in step 1.1 integrates the BiFPN network and the TensorRT network based on the original YOLOv8 model.

Images