Checking method, checking system and AR (Augmented Reality) equipment

Abstract

The invention relates to a checking method, a checking system and AR (Augmented Reality) equipment. The inspection method comprises the following steps: generating a scanning image of an inspected vehicle entering a corresponding inspection site through scanning equipment arranged in each inspection site of one or more inspection sites; entity inspection is carried out on inspected vehicles entering the corresponding inspection sites through AR equipment carried by workers of the inspection sites, wherein entity inspection comprises the steps that vehicle attribute information of the inspected vehicles is determined; classifying the detected vehicle based on the scanned image from the scanning device and the vehicle attribute information; and based on the determined category of the detected vehicle, performing physical inspection on the detected vehicle to determine whether the detected vehicle should be released or not.

Description

Technical Field

[0001] The present application relates to the field of security inspection technology, and in particular to an inspection method, an inspection system and an AR device. Background Art

[0002] In the existing inspection system, the inspection equipment in the central control room near the inspection site or the inspection equipment far from the inspection site instructs the inspection site staff to inspect the inspected vehicle through instructions. The inspection site staff can only passively follow the instructions and cannot conduct timely and effective inspections in real time according to the on-site situation. Summary of the invention

[0003] According to one aspect of the present application, an inspection method is provided, comprising: generating a scanned image of an inspected vehicle entering a corresponding inspection site by means of a scanning device arranged at each of one or more inspection sites; performing a physical inspection on the inspected vehicle entering the corresponding inspection site by means of an AR device carried by staff at each inspection site, wherein the physical inspection comprises: determining vehicle attribute information of the inspected vehicle; classifying the inspected vehicle based on the scanned image and the vehicle attribute information from the scanning device; and performing a physical inspection on the inspected vehicle based on the determined category of the inspected vehicle to determine whether the inspected vehicle should be released.

[0004] According to one aspect of the present application, an inspection system is provided, comprising: one or more inspection sites, each inspection site comprising: a scanning device configured to generate a scanned image of an inspected vehicle entering the corresponding inspection site; and an AR device carried by a staff member of the corresponding inspection site and configured to perform a physical inspection on the inspected vehicle, wherein the physical inspection comprises: determining vehicle attribute information of the inspected vehicle; classifying the inspected vehicle based on the scanned image and the vehicle attribute information from the scanning device; and based on the determined category of the inspected vehicle, performing a physical inspection on the inspected vehicle to determine whether the inspected vehicle should be released.

[0005] According to one aspect of the present application, an inspection method is provided, comprising: determining vehicle attribute information of an inspected vehicle entering an inspection site; classifying the inspected vehicle based on a scanned image and vehicle attribute information obtained by scanning the inspected vehicle with a scanning device at the inspection site; and based on the determined category of the inspected vehicle, performing a physical inspection of the inspected vehicle to determine whether the inspected vehicle should be released.

[0006] According to one aspect of the present application, an AR device is provided, comprising: a vehicle attribute information determination unit, configured to determine the vehicle attribute information of an inspected vehicle entering an inspection site; a vehicle classification unit, configured to classify the inspected vehicle based on a scanned image and vehicle attribute information obtained by scanning the inspected vehicle with a scanning device at the inspection site; and an inspection unit, configured to perform a physical inspection on the inspected vehicle based on the determined category of the inspected vehicle to determine whether the inspected vehicle should be released.

[0007] According to the inspection method, inspection system and AR device of the embodiments of the present application, the inspection site staff can actively conduct physical inspections of the inspected vehicles through the AR devices they carry, thereby enabling timely and effective inspections to be carried out in real time and dynamically. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present application in any way. In the accompanying drawings:

[0009] Figure 1 A network diagram of a verification system according to an embodiment of the present application is shown;

[0010] Figure 2 A flow chart showing a verification method according to an embodiment of the present application is shown;

[0011] Figure 3 A flowchart of performing entity inspection according to an embodiment of the present application is shown;

[0012] Figure 4 A schematic block diagram of an AR device according to an embodiment of the present application is shown;

[0013] Figure 5 A schematic diagram showing a blank vehicle image according to an embodiment of the present application;

[0014] Figure 6 A schematic diagram showing object identification according to an embodiment of the present application;

[0015] Figure 7 A flow chart showing a verification method according to an embodiment of the present application is shown;

[0016] Figure 8 A flow chart showing the determination of vehicle attributes of a detected vehicle according to an embodiment of the present application;

[0017] Fig. 9 A flowchart showing execution of entity inspection according to an embodiment of the present application is shown;

[0018] Fig.10 A schematic block diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] As mentioned above, in the existing inspection system, the inspection site staff can only passively follow the command and cannot conduct timely and effective inspections in real time and dynamically according to the on-site situation.

[0020] In view of this, the present application provides an inspection method, an inspection system and an AR device, so that inspection site staff can actively conduct physical inspections of the inspected vehicles through the AR devices they carry, thereby enabling timely and effective inspections to be carried out in real time and dynamically.

[0021] Figure 1 is a network diagram showing a verification system according to an embodiment of the present application. Figure 1 As shown, the inspection system 100 according to the embodiment of the present application includes one or more inspection sites, each of which includes:

[0022] A scanning device 101 is configured to generate a scanned image of the inspected vehicle entering the corresponding inspection site; and

[0023] The AR device 102 is carried by a staff member at the corresponding inspection site and is configured to perform a physical inspection on the inspected vehicle, wherein the physical inspection includes:

[0024] Determining vehicle attribute information of the inspected vehicle;

[0025] classifying the inspected vehicle based on the scanned image from the scanning device and the vehicle attribute information; and

[0026] Based on the determined category of the inspected vehicle, a physical inspection is performed on the inspected vehicle to determine whether the inspected vehicle should be released.

[0027] In some implementations, according to the inspection system 100 of the embodiment of the present application, each inspection site may further include:

[0028] The primary inspection device 103 is arranged near the corresponding inspection site, and is configured to receive the scanned image from the scanning device 102 at the corresponding inspection site, analyze the scanned image to provide a local inspection result.

[0029] In some implementations, the inspection system 100 according to the embodiment of the present application may also include a secondary inspection device 104, which is arranged remotely from one or more inspection sites and is configured to respond to scanned images sent from the primary inspection device at each inspection site, analyze the scanned images to provide remote inspection results and feed back to the primary inspection device 103 at the corresponding inspection site.

[0030] In some implementations, the AR device 101 may be configured to receive a local inspection result from the primary inspection device 103 at the corresponding inspection site or a remote inspection result from a distant secondary inspection device, and perform a physical inspection when the local inspection result or the remote inspection result indicates that the inspected vehicle has a suspicious area. It should be understood that the AR device 101 may perform the physical inspection on its own, that is, it does not rely on the local inspection result or the remote inspection result, or it may perform the physical inspection only when the local inspection result or the remote inspection result indicates that the inspected vehicle has a suspicious area, and it may be flexibly arranged depending on the needs of the workflow.

[0031] In some implementations, the AR device 101 may be configured to obtain real-time guidance from a primary inspector at the primary inspection device 103 at the corresponding inspection site or a secondary inspector at a remote secondary inspection device 104 during the execution of the entity inspection.

[0032] Specifically, in an embodiment according to the present application, there may be one or more inspection sites, such as a port site. Figure 1 The network shows inspection site 1 ... inspection site N. The ellipsis between inspection site 1 and inspection site N indicates that there may be one or more additional inspection sites between the two. In other words, the number of inspection sites is not limited, and an appropriate number of inspection sites can be set according to the actual application environment.

[0033] In the embodiment of the present application, an AR device 101 is deployed for each inspection site, and the AR device is managed by the corresponding on-site inspector. The on-site inspector can perform a physical inspection of the inspected vehicle through the AR device 101. As an example, the AR device can be various smart wearable devices, such as AR glasses.

[0034] In an embodiment according to the present application, for each inspection site, a scanning device 102 is deployed. For example, the scanning device 102 can be installed and deployed at the inspection site to scan the inspected vehicle (such as a passenger car, bus or other vehicle or a transport vehicle such as a container), generate a scanned image, and send the scanned image and corresponding information (such as a license plate number, etc.) to the AR device. In an embodiment according to the present application, for each inspection site, a first-level inspection device 103 is deployed near the inspection site (also referred to as the central control room), and the first-level inspection device 103 is responsible for the corresponding inspection personnel, who can use their professional knowledge to make inspection judgments in combination with the processing results of the first-level inspection device 103. For each inspection site, a server 104 corresponding to the first-level inspection device 103 is also deployed. The first-level inspection device 103 can be in the form of a PC computer. The first-level inspection device 103 can receive the scanned image from the scanning device 102 through the server 104, and then analyze the scanned image to provide a local inspection result. The scanned image can be stored in the server 104 in association with the local inspection result, for example, to form a local vehicle inspection library. The scanned image and the local inspection result can also be sent to the corresponding AR device 101, so that the AR device 101 can perform a physical inspection only when the inspection result indicates that there is a suspicious area in the inspected vehicle, thereby improving the inspection efficiency and saving resources. The first-level inspection device 103 can also establish a video connection with the AR device, and provide inspection guidance through the real-time first-person perspective image transmitted back by the AR device.

[0035] The primary inspection device 103 may also be coupled to an inspection center via a server 104. The inspection center is generally located in an area far from the inspection site. Figure 1 As shown, the inspection center has a secondary inspection device 105, which is managed by a corresponding inspection personnel (called a remote expert), and the inspection expert can use his or her senior professional knowledge to make inspection judgments in combination with the processing results of the secondary inspection device 105. In an embodiment according to the present application, there may be one or more secondary inspection devices 105. Figure 1 Three secondary inspection devices 105 are shown, and it should be understood that more or less secondary inspection devices 105 and corresponding remote experts can be deployed, as shown by the ellipsis in the figure. The inspection center also deploys a server 106 associated with each secondary inspection device 105.

[0036] Each first-level inspection device 103 can send the scanned image to the server 106 of the inspection center in the form of an inspection task through the server 104 associated with it. The server 106 assigns the inspection task to one or more second-level inspection devices 105. The second-level inspection device 105 analyzes the scanned image to provide a remote inspection result and feeds the remote inspection result back to the first-level inspection device 103 via the server 106 and the server 104 as the inspection result. The inspection center aggregates the scanned images of all inspection sites (for example, stores them in the service 106, for example, to form a global vehicle inspection library), is responsible for the inspection operations of all inspection sites, and conducts unified management, guidance, and patrol command. Remote experts can conduct spot checks, analysis, and supervision and management of inspections at all levels through the second-level inspection device 105 according to risk level, timeliness, etc.

[0037] Generally speaking, the inspection experts responsible for the secondary inspection equipment 105 have richer and more senior professional knowledge, stronger image analysis capabilities and more experience knowledge than the inspectors responsible for the primary inspection equipment 103. Therefore, when the inspectors of the primary inspection equipment 103 encounter difficult problems, they can request the remote inspection of the inspection experts of the secondary inspection equipment 105 to provide more accurate inspection results.

[0038] The primary inspection device 103 at the inspection site and the secondary inspection device 105 at the inspection center can be computer devices commonly used in the art, and their types are not limited, as long as they have image processing and recognition capabilities and audio and video communication capabilities (installed with corresponding applications). In some implementations, the secondary inspection device 105 at the inspection center can have more powerful image recognition and processing capabilities than the primary inspection device 103 at the inspection site.

[0039] The server 104 at the inspection site and the server 106 at the inspection center can be any server commonly used in the art, and the type of the server is not limited, as long as it has certain data exchange and processing capabilities. In the case of several inspection sites, the server 106 at the inspection center needs to have a massive data processing capability.

[0040] In the embodiments according to the present application, in some implementations, the AR device 101 may also request real-time guidance from the primary inspection device 103 and the secondary inspection device 105 via video during the physical inspection (for example, when encountering difficult problems) to provide more accurate inspection results. In some implementations, the primary inspection device 103 and the secondary inspection device 105 may also temporarily or regularly send inspection requests to the AR device 101, and guide the physical inspection of the AR device 101 by establishing video communication with the AR device 101, so as to provide inspection management and supervision.

[0041] Figure 2FIG. 1 is a flow chart showing a method for checking according to an embodiment of the present application. Figure 2 As shown, according to the inspection method 200 of the embodiment of the present application, Figure 1 The inspection system shown is implemented, including steps S201-S202.

[0042] In step S201, a scanning image of a vehicle under inspection entering a corresponding inspection site is generated by a scanning device arranged at each inspection site of one or more inspection sites.

[0043] In step S202, the inspected vehicles at the corresponding inspection sites are physically inspected by AR devices carried by staff at the inspection sites.

[0044] The entity check performed in step S202 is Figure 3 As shown in Figure 3 As shown, performing entity inspection includes steps S301-S303.

[0045] In step S301, the vehicle attribute information of the inspected vehicle is determined.

[0046] In step S302, the inspected vehicle is classified based on the scanned image from the scanning device and the vehicle attribute information.

[0047] In step S303, based on the determined category of the inspected vehicle, a physical inspection is performed on the inspected vehicle to determine whether the inspected vehicle should be released.

[0048] In some implementations, the inspection method 200 according to the embodiment of the present application may also include: receiving scanned images from a scanning device at the corresponding inspection site by a first-level inspection device arranged near each inspection site, analyzing the scanned images to provide local inspection results; and responding to the scanned images sent by the first-level inspection device at each inspection site by a second-level inspection device arranged far away from each inspection site, analyzing the scanned images to provide remote inspection results and feeding back to the first-level inspection device at the corresponding inspection site.

[0049] In some implementations, the inspection method 200 according to the embodiment of the present application may also include: receiving, through the AR device at each inspection site, the local inspection results from the first-level inspection device of the corresponding inspection site or the remote inspection results from the second-level inspection device at a distance, wherein the physical inspection is performed when the local inspection result or the remote inspection result indicates that there is a suspicious area in the inspected vehicle.

[0050] In some implementations, the inspection method 200 according to the embodiment of the present application may also include: obtaining real-time guidance from a first-level inspector at the first-level inspection device at the corresponding inspection site or a second-level inspection personnel at a remote second-level inspection device during the execution of the entity inspection through the AR device at each inspection site.

[0051] According to the inspection system and inspection method of this application, the inspection site staff can actively conduct physical inspections of the inspected vehicles through the AR devices they carry, making full use of the intuitiveness and portability of the AR devices, so as to conduct timely and effective inspections in real time and dynamically. The inspection site staff can also seek remote assistance or receive additional inspection guidance through the AR devices when encountering difficult problems, making full use of the interactivity of the AR devices, establishing an inspection system that links up and down, and improving the accuracy and efficiency of the inspections.

[0052] Figure 4 A schematic block diagram of an AR device according to an embodiment of the present application is shown. Figure 4 As shown, the AR device 400 according to an embodiment of the present application includes:

[0053] The vehicle attribute information determination unit 401 is configured to determine the vehicle attribute information of the inspected vehicle entering the inspection site;

[0054] The vehicle classification unit 402 is configured to classify the inspected vehicle based on the scanned image and vehicle attribute information obtained by scanning the inspected vehicle by the scanning device at the inspection site; and

[0055] The inspection unit 403 is configured to perform a physical inspection on the inspected vehicle based on the determined category of the inspected vehicle to determine whether the inspected vehicle should be released.

[0056] In some implementations, the vehicle attribute information determination unit 401 may include:

[0057] The video capture unit 4011 is configured to capture real-time image frames of the inspected vehicle at a specific frame rate;

[0058] The image processing unit 4012 is configured to:

[0059] Preprocessing the captured image frames;

[0060] compressing the preprocessed image frame to obtain a compressed image; and

[0061] The determination unit 4013 is configured to determine the vehicle attribute information based on the compressed image and using a vehicle attribute recognition toolkit.

[0062] As an example, an AR device uses a built-in camera or other sensors to collect images and corresponding data of surrounding vehicles, and can implement real-time video preview to display the captured images. During the preview process, real-time image frame capture, image data preprocessing, image compression and other links are implemented. For example, during preview, the device captures image frames at a frequency of 60 frames per second. These frame data are continuous to ensure that users can see smooth images. The captured frame data is usually stored in the original format for subsequent processing. It should be understood that a frequency of 60 frames per second can provide a smoother visual experience, but it may also increase the burden of data processing and transmission. Therefore, the frame rate can be dynamically adjusted according to the network bandwidth or processing power to balance quality and performance.

[0063] In some implementations, performing preprocessing may include: performing at least one of contrast adjustment, brightness adjustment, denoising, and color correction. For example, for contrast adjustment, histogram equalization and adaptive histogram equalization (CLAHE) image processing algorithms may be used to change the brightness range of an image, enhance the difference between light and dark parts of an image, and make details more prominent. For brightness adjustment, the overall brightness of an image may be increased or decreased, making the image easier to observe under different lighting conditions. For denoising, mean filtering denoising techniques (smoothing images by calculating the average value of neighborhood pixels to reduce random noise) may be used to eliminate unnecessary noise in an image to improve image quality. For color correction, a white balance adjustment color correction algorithm may be used to adjust the color temperature of an image so that white objects in an image look real, thereby improving overall color performance. These processes may improve image quality, making subsequent analysis or display effects better.

[0064] In some implementations, performing image compression may include using the Luban algorithm to significantly reduce the file size while maintaining high image quality, reducing the storage resource requirements of the AR device. In some implementations, the vehicle attribute recognition toolkit (SDK) is called to enable the recognition and analysis of vehicle attributes. In some implementations, the vehicle attribute information includes the license plate number, brand, model, year, color, and similarity to historical risk vehicles of the inspected vehicle. In some implementations, the vehicle attribute information can be displayed on the display screen of the AR device in an image, text, or other manner.

[0065] Through AR equipment, vehicle attributes can be quickly identified to obtain information such as license plate number, brand, model, year, color, and similarity with historical risk vehicles, thereby reducing manual entry steps, classifying risks through diversified data, assisting on-site inspections, and improving inspection accuracy.

[0066] In some implementations, the vehicle classification unit 402 may be configured to classify the inspected vehicle into one of the following categories: a scanned vehicle, or an unscanned vehicle.

[0067] In some implementations, the verification unit 403 may include:

[0068] A comparison unit 4031, wherein the comparison unit 4031 is configured to: compare the scanned image of the inspected vehicle with the body of the inspected vehicle when the inspected vehicle is determined to be a scanned vehicle;

[0069] The determination unit 4032 makes a check result based on the comparison result.

[0070] In some implementations, the checking unit 403 may further include:

[0071] A blank image generating unit 4033, the blank image generating unit 4033 is configured to: generate a blank vehicle image based on the vehicle attribute information when the inspected vehicle is determined to be an unscanned vehicle, wherein the blank vehicle image is segmented into two or more regions corresponding to respective parts of the body of the inspected vehicle;

[0072] The comparison unit 4031 is configured to compare the generated blank vehicle image having two or more regions with the body of the inspected vehicle; and

[0073] The determination unit 4032 is configured to make a verification result based on the comparison result.

[0074] If the scanning equipment is under maintenance or there is no scanning equipment at the inspection site, the AR device supports inspection and analysis of non-scanned images. Figure 5 A schematic diagram of a blank vehicle image is shown in FIG. 5 , according to the vehicle model (such as a sedan, SUV, MPV, etc.), the vehicle body is divided into the front engine area 1, the co-pilot area 2, the right passenger area 3, the main cab area 4, the left passenger area 5, the rear area 6, etc. It should be understood that this division is only schematic, and other division methods can also be used. For other vehicles, other division methods can also be used. After the vehicle body is divided into areas, physical comparisons can be quickly performed on these areas to make inspection results.

[0075] In some implementations, the determining unit 4032 may further include:

[0076] The video capture unit 4032-1 is configured to capture an image of a suspicious object when the inspection result indicates that the inspected vehicle has a suspicious object;

[0077] The object identification unit 4032-2 is configured to: perform object identification on the suspicious object from the image; and

[0078] The first information acquisition unit 4032-3 is configured to acquire reference information corresponding to the object based on the identified object, wherein the inspection unit determines whether the inspected vehicle should be released based on the reference information.

[0079] For example, suspicious items include alcohol, cigarettes, etc. Reference information includes relevant laws and regulations and tax-related information.

[0080] In some implementations, capturing an image of a suspicious object may include: capturing real-time image frames of the suspicious object at a specific frame rate; and preprocessing the captured image frames.

[0081] As an example, an AR device uses a built-in camera or other sensors to collect images and corresponding data of surrounding vehicles, and can realize real-time video preview to display the captured images. In the preview process, image capture, image data preprocessing and other links are implemented. For example, during preview, the device captures image frames at a frequency of 60 frames per second. These frame data are continuous, ensuring that users can see smooth images. The captured frame data is usually stored in the original format for subsequent processing. It should be understood that a frequency of 60 frames per second can provide a smoother visual experience, but it may also increase the burden of data processing and transmission. Therefore, the frame rate can be dynamically adjusted according to the network bandwidth or processing power to balance quality and performance. After each frame of the image is acquired, image preprocessing is usually performed, such as contrast adjustment, brightness adjustment, and color correction. These processes can improve the image quality, making the subsequent analysis or display effect better.

[0082] In some implementations, the item identification unit 4032-2 may be configured as follows: Figure 6 shown.

[0083] The object identification unit 4032-2 may be configured as follows:

[0084] detecting a subject of the suspicious object from the image to determine a subject image;

[0085] Extracting features from the subject image with reference to a preset image library to obtain a feature vector; and

[0086] Based on the feature vector, a preset feature vector having the highest similarity with the feature vector is retrieved from a preset feature vector library and an item corresponding to the preset feature vector is determined.

[0087] In the embodiment of the present application, computer vision technology, such as subject detection, feature extraction, vector search, etc., is used to identify objects. For subject detection, subject detection is performed on the image of the object to be identified, the coordinate position of one or more subjects in the image is detected, and then the corresponding area in the image is cropped for identification. Subject detection is a pre-step of the recognition task. The input image is identified after subject detection, which can filter complex backgrounds and effectively improve the recognition accuracy. The subject detection model used by the AR device requires fast speed, high accuracy, small size, and friendly deployment environment. As an example, the lightweight model PicoDet can be selected. For feature extraction, it is a key link in image recognition. Its function is to convert the input image into a feature vector of fixed dimension for subsequent vector retrieval. Considering the speed, model size, feature extraction performance and other factors of the feature extraction model, as an example, PPLCNetV2_base can be selected as the feature extraction network. The main goal of vector retrieval is to perform similarity or distance calculation of feature vectors in the established vector library for a given query vector, and return the similarity ranking result of the candidate vectors. Considering the advantages of good adaptability, easy installation, rich algorithms, and support for both CPU and GPU, as an example, the Faiss vector retrieval open source library can be selected.

[0088] It should be understood that according to an embodiment of the present application, an image captured from a suspicious area may contain more than one object, and at least one subject image can be determined in subject detection, and feature vectors of each subject image can be extracted and vector retrieval can be performed on each feature vector to identify more than one object.

[0089] In some implementations, the threshold range of recognition can be set according to the needs and scene characteristics of different object recognition. For example, the confidence threshold of alcoholic objects is set to 80%. Only when the confidence of the recognition result is higher than the threshold, it is considered to be a valid recognition. At the same time, the accuracy and speed of the recognition results are monitored, and the threshold is adjusted reasonably according to the monitoring results.

[0090] Through the object recognition function provided by AR devices, reference information such as taxable goods and corresponding legal and regulatory knowledge can be obtained during the inspection process to assist on-site inspectors in making comprehensive decisions.

[0091] Return to reference Figure 4 In some implementations, the AR device 400 may further include: a communication unit 404, configured to:

[0092] During a physical inspection, proactively establish a video connection with a primary inspection device of a primary inspector near the inspection site or a secondary inspection device of a secondary inspector remote from the inspection site;

[0093] Transmitting the first-person perspective image to the primary inspection device or the secondary inspection device; and

[0094] Receive real-time guidance based on the first-perspective picture from a first-level inspector via a first-level inspection device or real-time guidance based on the first-perspective picture from a second-level inspector via a second-level inspection device.

[0095] In some implementations, the communication unit 404 may be configured to:

[0096] Receive regular or ad hoc inspection requests for primary inspection equipment from primary inspectors or secondary inspection equipment from secondary inspectors; and

[0097] In response to the inspection application, a video connection is established with a primary inspection device of a primary inspector or a secondary inspection device of a secondary inspector.

[0098] In some implementations, the communication unit 404 may be configured to establish a point-to-point video connection with a primary inspection device of a primary inspector or a secondary inspection device of a secondary inspector through a network. As an example, a WebRTC-based real-time audio and video call may be used.

[0099] AR devices integrate real-time audio and video call functions, allowing users to communicate and guide remotely with inspectors in the central control room or remote experts in the inspection center, and realize screen sharing functions, so that inspectors can see the user's perspective and provide accurate guidance. Inspectors in the central control room or remote experts in the inspection center can fully understand the on-site operation situation and establish a sound supervision and inspection system.

[0100] In some implementations, the AR device 400 may further include a historical verification information acquisition unit 405, and the historical verification information acquisition unit 405 includes:

[0101] The speech recognition unit 4051 is configured to: receive speech input from on-site personnel during entity inspection; and

[0102] The second information acquisition unit 4052 obtains historical inspection information from a preset knowledge base in response to the voice input, wherein determining whether the inspected vehicle should be released is also based on the historical inspection information.

[0103] The preset knowledge base may be, for example, a local vehicle inspection base stored in the server 104 and a remote vehicle inspection base stored in the server 106. Through the voice interaction function provided by the AR device, historical vehicle inspection information is obtained during the inspection process to assist on-site inspectors in making comprehensive decisions.

[0104] According to the inspection system of this application, the inspection site staff can actively conduct physical inspections of the inspected vehicles through the AR devices they carry, making full use of the intuitiveness and portability of the AR devices, so as to conduct timely and effective inspections in real time and dynamically. The inspection site staff can also seek remote assistance or receive additional inspection guidance through the AR devices when encountering difficult problems, making full use of the interactivity of the AR devices, establishing an inspection system that links up and down, and improving the accuracy and efficiency of the inspection.

[0105] Figure 7 FIG. 7 is a flow chart showing a verification method according to an embodiment of the present application. The verification method 700 according to an embodiment of the present application is executed by an AR device according to an embodiment of the present application. Figure 7 As shown, the inspection method 700 according to the embodiment of the present application includes S701-S703.

[0106] In step S701, the vehicle attribute information of the inspected vehicle entering the inspection site is determined;

[0107] In step S702, the inspected vehicle is classified based on the scanned image and vehicle attribute information obtained by scanning the inspected vehicle with the scanning equipment at the inspection site; and

[0108] In step S703, based on the determined category of the inspected vehicle, a physical inspection is performed on the inspected vehicle to determine whether the inspected vehicle should be released.

[0109] In some implementations, determining the vehicle attribute information of the inspected vehicle in step S701 may include steps S701-1-S701-4, such as Figure 8 shown.

[0110] In step S701-1, real-time image frames are captured for the inspected vehicle at a specific frame rate;

[0111] In step S701-2, the captured image frame is preprocessed;

[0112] In step S701-3, compressing the preprocessed image frame to obtain a compressed image; and

[0113] In step S701 - 4 , vehicle attribute information is determined based on the compressed image and using a vehicle attribute recognition toolkit.

[0114] In some implementations, classifying the inspected vehicle in step S702 may include classifying the inspected vehicle into one of the following categories: a scanned vehicle and an unscanned vehicle.

[0115] In some implementations, the entity check is performed in step S703 such as Figure 7 As shown. Figure 7As shown, when the inspected vehicle is determined to be a scanned vehicle (with a scanned image), step S703 includes:

[0116] Step S7031, comparing the scanned image of the inspected vehicle with the body of the inspected vehicle; and

[0117] Step S7032, determine the inspection result based on the comparison result.

[0118] like Figure 7 As shown, when the inspected vehicle is determined to be an unscanned vehicle (no scanned image), step S703 includes:

[0119] Step S7033, generating a blank vehicle image based on the vehicle attribute information, wherein the blank vehicle image is divided into two or more regions corresponding to various parts of the body of the inspected vehicle;

[0120] Step S7034, comparing the generated blank vehicle image having two or more regions with the body of the inspected vehicle; and

[0121] Step S7035, determine the inspection result based on the comparison result.

[0122] In some implementations, when the inspection result indicates that the inspected vehicle has a suspicious object, step S703 further includes:

[0123] Step S7036, capturing an image of the suspicious object;

[0124] Step S7037, performing object identification on the suspicious object; and

[0125] Step S7038, obtaining reference information corresponding to the object based on the identified object, wherein determining whether the inspected vehicle should be released is also based on the reference information.

[0126] In some implementations, performing item identification in step S7037 may include:

[0127] detecting a subject of the suspicious object from the image to determine a subject image;

[0128] Extracting features from the subject image with reference to a preset image library to obtain a feature vector; and

[0129] Based on the feature vector, a preset feature vector having the highest similarity with the feature vector is retrieved from a preset feature vector library and an item corresponding to the preset feature vector is determined.

[0130] In some implementations, the verification method 700 according to the embodiment of the present application may further include:

[0131] During a physical inspection, proactively establish a video connection with a primary inspection device of a primary inspector near the inspection site or a secondary inspection device of a secondary inspector remote from the inspection site;

[0132] Transmitting the first-person perspective image to the primary inspection device or the secondary inspection device; and

[0133] Receive real-time guidance based on the first-perspective picture from a first-level inspector via a first-level inspection device or real-time guidance based on the first-perspective picture from a second-level inspector via a second-level inspection device.

[0134] In some implementations, the verification method 700 according to the embodiment of the present application may further include:

[0135] Receive regular or ad hoc inspection requests for primary inspection equipment from primary inspectors or secondary inspection equipment from secondary inspectors; and

[0136] In response to the inspection application, a video connection is established with a primary inspection device of a primary inspector or a secondary inspection device of a secondary inspector.

[0137] In some implementations, the video connection may be a point-to-point video connection through a network penetration with a primary inspection device of a primary inspector or a secondary inspection device of a secondary inspector.

[0138] In some implementations, the inspection method 700 according to the embodiment of the present application may also include: during the physical inspection, obtaining historical inspection information from a preset knowledge base in response to voice input from on-site personnel, wherein determining whether the inspected vehicle should be released is also based on the historical inspection information.

[0139] According to the inspection system of this application, the inspection site staff can actively conduct physical inspections of the inspected vehicles through the AR devices they carry, making full use of the intuitiveness and portability of the AR devices, so as to conduct timely and effective inspections in real time and dynamically. The inspection site staff can also seek remote assistance or receive additional inspection guidance through the AR devices when encountering difficult problems, making full use of the interactivity of the AR devices, establishing an inspection system that links up and down, and improving the accuracy and efficiency of the inspection.

[0140] Fig.10 A schematic block diagram of a computer device that can be used to implement the inspection device according to an embodiment of the present application is shown. Fig.10As shown, the computer device 1000 according to an embodiment of the present application may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. Various programs and data required for the operation of the computer device 1000 are also stored in RAM 1004. The processing device 1001, ROM 1002, storage device 1003 and RAM 1004 are connected to each other via a bus 1005. A communication device 1006 is also connected to the bus 1005. It can be coupled and communicated with external devices via the communication device 1006. It should be understood that. It should be understood that Fig.10 The block diagram shown is only an example to provide an understanding of the embodiments of the present application, and there may be more or fewer components.

[0141] Although Fig.10 The computer device 1000 is shown with various components, but it should be understood that it is not required to implement or have all of the components shown. More or fewer components may be implemented or have instead. Fig.10 Each box shown in the figure may represent one component or multiple components as required.

[0142] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure provides a computer-readable storage medium that stores a computer program, which contains program code for executing the inspection method according to an embodiment of the present application. In such an embodiment, the computer program can be installed from the storage device 1003, or installed from the ROM 1002, or downloaded and installed from the network. When the computer program is executed by the processing device 1001, the inspection method according to the embodiment of the present application is implemented.

[0143] It should be noted that the computer-readable medium according to an embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium according to an embodiment of the present invention may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device.

[0144] Computer program code for performing operations according to embodiments of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages.

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions, and operations of the systems and methods according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, which contains one or more executable instructions for implementing a specified logical function.

[0146] The detailed description of the embodiments of the present invention covers many specific details in order to provide a comprehensive understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without the need for some of these specific details. The above description of the embodiments is only to provide a clearer understanding of the present invention by illustrating examples of the present invention. The present invention is by no means limited to any specific configuration and method steps proposed below, but covers any modification, replacement and improvement of the relevant elements, parts and method steps without departing from the teachings of the present invention.

[0147] It should be noted that, in the claims, the word "comprise" or "comprising" does not exclude the presence of elements or components not listed in the claims. The article "a" or "an" preceding an element or component does not exclude the presence of a plurality of such elements or components.

[0148] In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or limiting the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. With respect to the scope of the present application, the descriptions made in the specification are illustrative rather than restrictive, and the scope of the present application is defined by the appended claims.

Claims

1. A verification method, comprising: Generate a scanned image of the inspected vehicle entering the corresponding inspection site by means of a scanning device arranged at each inspection site of one or more inspection sites; The inspected vehicles entering the corresponding inspection site are physically inspected by AR devices carried by the staff at each inspection site, wherein the physical inspection includes: Determining vehicle attribute information of the inspected vehicle; classifying the inspected vehicle based on the scanned image from the scanning device and the vehicle attribute information; and Based on the determined category of the inspected vehicle, a physical inspection is performed on the inspected vehicle to determine whether the inspected vehicle should be released.

2. The inspection method according to claim 1, further comprising: Receiving the scanned image from the scanning device at the corresponding inspection site through the primary inspection device arranged near each inspection site, analyzing the scanned image to provide a local inspection result; as well as The secondary inspection device arranged at a distance from each inspection site responds to the scanned image sent from the primary inspection device at each inspection site, analyzes the scanned image to provide a remote inspection result and feeds it back to the primary inspection device at the corresponding inspection site.

3. The inspection method according to claim 2, further comprising: The local inspection result from the first-level inspection device at the corresponding inspection site or the remote inspection result from the second-level inspection device at a distance is received through the AR device at each inspection site, wherein the physical inspection is performed when the local inspection result or the remote inspection result indicates that there is a suspicious area in the inspected vehicle.

4. The inspection method according to claim 2, further comprising: During the execution of the entity inspection, the AR device at each inspection site obtains real-time guidance from the primary inspector at the primary inspection device at the corresponding inspection site or the secondary inspector at the remote secondary inspection device.

5. An inspection system comprising: One or more inspection sites, each inspection site includes: A scanning device configured to generate a scanned image of the inspected vehicle entering the corresponding inspection site; The AR device is carried by a staff member at the corresponding inspection site and is configured to perform a physical inspection on the inspected vehicle, wherein the physical inspection includes: Determining vehicle attribute information of the inspected vehicle; classifying the inspected vehicle based on the scanned image from the scanning device and the vehicle attribute information; and Based on the determined category of the inspected vehicle, a physical inspection is performed on the inspected vehicle to determine whether the inspected vehicle should be released.

6. The inspection system according to claim 5, wherein: Each inspection site also includes: A primary inspection device is arranged near the corresponding inspection site and is configured to receive a scanned image from the scanning device at the corresponding inspection site and analyze the scanned image to provide a local inspection result; and The inspection system also includes a secondary inspection device, which is arranged remotely from the one or more inspection sites and is configured to respond to the scanned images sent from the primary inspection device at each inspection site, analyze the scanned images to provide remote inspection results and feed them back to the primary inspection device at the corresponding inspection site.

7. The inspection system according to claim 6, wherein: The AR device is configured to receive the local inspection result from the first-level inspection device at the corresponding inspection site or the remote inspection result from the second-level inspection device at a distance, and perform the physical inspection when the local inspection result or the remote inspection result indicates that there is a suspicious area in the inspected vehicle.

8. The inspection system according to claim 6, wherein: The AR device is configured to obtain real-time guidance from a primary inspector at the primary inspection device at a corresponding inspection site or a secondary inspector at a remote secondary inspection device during execution of the entity inspection.

9. A verification method comprising: Determine the vehicle attribute information of the inspected vehicle entering the inspection site; Classifying the inspected vehicle based on a scanned image obtained by scanning the inspected vehicle with a scanning device at the inspection site and the vehicle attribute information; Based on the determined category of the inspected vehicle, a physical inspection is performed on the inspected vehicle to determine whether the inspected vehicle should be released.

10. The inspection method according to claim 9, wherein determining the vehicle attribute information of the inspected vehicle comprises: Capturing real-time image frames of the inspected vehicle at a specific frame rate; Preprocessing the captured image frames; Compressing the preprocessed image frame to obtain a compressed image; as well as The vehicle attribute information is determined based on the compressed image and using a vehicle attribute recognition toolkit.

11. The inspection method according to claim 9, wherein classifying the inspected vehicle comprises classifying the inspected vehicle into one of the following categories: a scanned vehicle, and an unscanned vehicle.

12. The inspection method according to claim 11, wherein: The physical inspection of the inspected vehicle includes: When the inspected vehicle is determined to be the scanned vehicle, comparing the scanned image of the inspected vehicle with the body of the inspected vehicle; and A verification result is made based on the result of the comparison.

13. The inspection method according to claim 11, wherein: The physical inspection of the inspected vehicle includes: If the inspected vehicle is determined to be the unscanned vehicle, generating a blank vehicle image based on the vehicle attribute information, wherein the blank vehicle image is segmented into two or more regions corresponding to respective parts of the body of the inspected vehicle; and comparing the generated blank vehicle image having two or more regions with the body of the inspected vehicle; and A verification result is made based on the result of the comparison.

14. The inspection method according to claim 12 or 13, further comprising: When the inspection result indicates that there is a suspicious object on the inspected vehicle, capturing an image of the suspicious object; Performing object identification on the suspicious object; as well as Reference information corresponding to the object is acquired based on the identified object, wherein determining whether the inspected vehicle should be released is also based on the reference information.

15. The verification method of claim 14, wherein performing item identification comprises: detecting a subject of the suspicious object from the image to determine a subject image; Extracting features from the subject image with reference to a preset image library to obtain a feature vector; as well as Based on the feature vector, a preset feature vector having the highest similarity with the feature vector is retrieved from a preset feature vector library and an item corresponding to the preset feature vector is determined.

16. The inspection method according to claim 9, further comprising: During the physical inspection, actively establish a video connection with a primary inspection device near the inspection site or a secondary inspection device far from the inspection site; Transmitting the first-perspective image to the first-level inspection device or the second-level inspection device; as well as Receive real-time guidance based on the first perspective picture from the first-level inspector via the first-level inspection device or receive real-time guidance based on the first perspective picture from the second-level inspector via the second-level inspection device.

17. The inspection method according to claim 16, further comprising: Receiving regular or temporary inspection applications for the first-level inspection equipment from the first-level inspectors or the second-level inspection equipment from the second-level inspectors; as well as A video connection is established with the primary inspection device of the primary inspection personnel or the secondary inspection device of the secondary inspection personnel in response to the inspection application.

18. The inspection method according to claim 9, further comprising: During the physical inspection, historical verification information is obtained from a preset knowledge base in response to voice input from on-site personnel, wherein determining whether the inspected vehicle should be released is also based on the historical verification information.

19. An AR device, comprising: A vehicle attribute information determination unit configured to determine vehicle attribute information of a vehicle under inspection entering the inspection site; A vehicle classification unit configured to classify the inspected vehicle based on a scanned image obtained by scanning the inspected vehicle by a scanning device at the inspection site and the vehicle attribute information; as well as The checking unit is configured to perform a physical inspection on the checked vehicle based on the determined category of the checked vehicle to determine whether the checked vehicle should be released.

20. The AR device according to claim 19, wherein the vehicle attribute information determination unit comprises: A video capture unit is configured to capture real-time image frames of the inspected vehicle at a specific frame rate; The image processing unit is configured to: Preprocessing the captured image frames; Compressing the preprocessed image frame to obtain a compressed image; as well as The determining unit is configured to determine the vehicle attribute information based on the compressed image and using a vehicle attribute recognition toolkit. 21 . The AR device according to claim 19 , wherein the vehicle classification unit is configured to classify the inspected vehicle into one of the following categories: a scanned vehicle, an unscanned vehicle.

22. The AR device according to claim 21, wherein: The inspection unit comprises: a comparing unit configured to: compare the scanned image of the inspected vehicle with the body of the inspected vehicle when the inspected vehicle is determined to be the scanned vehicle; and The determination unit is configured to make a verification result based on the result of the comparison.

23. The AR device according to claim 22, wherein: The checking unit further includes a blank image generating unit configured to: In the case where the inspected vehicle is determined to be the unscanned vehicle, generating a blank vehicle image based on the vehicle attribute information, wherein the blank vehicle image is segmented into two or more regions corresponding to respective parts of the body of the inspected vehicle; and The comparison unit is configured to compare the generated blank vehicle image having two or more regions with the body of the inspected vehicle; and The determination unit is configured to make a verification result based on the result of the comparison.

24. The AR device according to claim 22 or 23, wherein the determining unit further comprises: A video capture unit is configured to capture an image of the suspicious object when the inspection result indicates that the inspected vehicle has a suspicious object; an object identification unit, configured to: perform object identification on the suspicious object from the image; as well as The first information acquisition unit is configured to acquire reference information corresponding to the object based on the identified object, wherein the checking unit determines whether the inspected vehicle should be released based on the reference information.

25. The AR device according to claim 24, wherein: The object identification unit is configured to: detecting a subject of the suspicious object from the image to determine a subject image; Extracting features from the subject image with reference to a preset image library to obtain a feature vector; and Based on the feature vector, a preset feature vector having the highest similarity with the feature vector is retrieved from a preset feature vector library and an item corresponding to the preset feature vector is determined.

26. The AR device according to claim 1, further comprising: The communication unit is configured as follows: During the physical inspection, actively establish a video connection with a primary inspection device near the inspection site or a secondary inspection device far from the inspection site; Transmitting the first-perspective image to the first-level inspection device or the second-level inspection device; as well as Receive real-time guidance from the first-level inspector of the first-level inspection device based on the first viewing angle picture or receive real-time guidance from the second-level inspector of the second-level inspection device based on the first viewing angle picture.

27. The AR device according to claim 26, wherein the communication unit is configured to: receiving regular or temporary inspection applications from the first-level inspector via the first-level inspection equipment or the second-level inspector via the second-level inspection equipment; and A video connection is established with the primary inspection device of the primary inspection personnel or the secondary inspection device of the secondary inspection personnel in response to the inspection application.

28. The AR device according to claim 19, further comprising a historical verification information acquisition unit, the historical verification information acquisition unit comprising: a speech recognition unit configured to: receive speech input from a field person during the entity inspection; as well as The second information acquisition unit obtains historical inspection information from a preset knowledge base in response to the voice input, wherein determining whether the inspected vehicle should be released is also based on the historical inspection information.

Images