Wearable intelligent AR (Augmented Reality) recognition glasses for customs

Abstract

The invention discloses wearable intelligent AR (Augmented Reality) recognition glasses for customs. The AR glasses comprise six main function modules, namely a think tank retrieval module, a remote cooperation module, a standard operation module, an AR recognition module, an intelligent identification module and an auxiliary translation module; the think tank retrieval comprises a forbidden directory function module and a case function module, the remote cooperation is a remote function auxiliary module, the standard operation module guides and standardizes the operation process according to the preset standard operation, and the AR identification comprises six function modules of illegal publications, endangered species, alien species, drugs, infringement and counterfeit, and gun parts. The intelligent identification mainly realizes a remote diagnosis and identification function. According to the system, the functions of real-time visualization, data acquisition, audio and video interaction and the like can be realized, the customs supervision work efficiency is improved, the AR glasses can be controlled through voice during use, the effect of liberating two hands is achieved, and front-line workers can carry out supervision work more conveniently.

Description

Technical Field

[0001] The present invention relates to the technical field of customs identification of "prohibited and restricted entry and exit articles", and specifically to a pair of wearable smart AR glasses for customs. Background Art

[0002] In the prior art, the daily supervision work of customs is complicated and tedious. In actual work, the identification and appraisal of various prohibited and restricted items, the memorization of the prohibited and restricted items list and operating procedures are usually completed by work experience and human brain memory. However, there are many related requirements and regulations, which are difficult to remember clearly. Relying solely on manual memory is bound to cause many errors. Therefore, there are various identification devices used for customs work in the prior art.

[0003] For example: the "A Customs Intelligent Identification and Management System" disclosed in the technical solution of Publication (Announcement) No. CN111767967A includes the following three components: a. Customs intelligent helmet device; b. Customs intelligent helmet cloud computing; c. System software, which is installed in the customs intelligent helmet device and the customs intelligent helmet cloud computing; the customs intelligent helmet device includes the following parts: microphone, gyroscope, speaker, camera, battery, image recognition, positioning system, voice recognition, communication system, sensor, display module, central processing unit and control module. The customs intelligent helmet device picks up external corresponding information through sensors such as microphone, gyroscope and camera, and performs image recognition, positioning system, voice recognition and positioning processing on the information through the central processing unit and control module. The customs intelligent helmet device of the present invention integrates multiple functional modules in one, which can effectively improve the work efficiency and safety of customs staff.

[0004] For example: the technical solution disclosed in the publication (announcement) number: CN210353391U discloses a "work hat for real-time monitoring of customs law enforcement based on AR technology". The technical solution includes a hat strap, a hat body, a microphone, a camera, a stand, a fill light, a customs logo and a data collection mechanism. The present invention, by setting the hat body, sets the hat shell to a hemispherical shape, which can have enough space to accommodate the data collection mechanism. The setting of the brim can prevent the microphone from getting wet on rainy days and prevent rain from entering the collar of the law enforcement officer; by setting the data collection mechanism, the single-chip microcomputer stores the data in the storage module, and transmits it to the customs monitoring terminal through the data transmission module, so that the office monitoring personnel can understand the law enforcement situation of the law enforcement personnel and stop some violent law enforcement behaviors in time; by setting the microphone and the camera, the microphone records the sound of the law enforcement scene, and the camera records the picture, which allows the customs monitoring terminal personnel to observe in an immersive way.

[0005] However, the above technical solution only discloses the equipment for customs identification, but does not mention the specific items for customs identification, the specific operating process standard settings, and how to accurately identify customs goods products.

[0006] Therefore, in order to solve the above problems, it is necessary to develop a wearable intelligent AR recognition glasses for customs that can improve work efficiency and quality. Summary of the invention

[0007] The purpose of the present invention is to provide a wearable intelligent AR recognition glasses for customs in view of the shortcomings of the prior art; the technical solution is as follows:

[0008] A wearable intelligent AR recognition glasses for customs, including a think tank retrieval module, an AR recognition module and a standard operation module;

[0009] The think tank search module includes a prohibited and restricted catalog module and a case module. The prohibited and restricted catalog module includes a catalog of dangerous chemicals, a catalog of psychotropic drugs, a catalog of narcotic drugs, the CITES Convention, a catalog of precursor chemicals, a list of prohibited animals and plants and products and other quarantine items, and other catalogs for customs inspection;

[0010] The AR recognition module has a shooting function and a voice input control function, and the types of recognition of the AR recognition module include illegal publications, endangered species, alien species, drugs, infringement and counterfeiting, and gun parts;

[0011] Use the voice input function of the AR glasses to select a module. After entering the module, use the AR glasses to move the object to be identified to the front of the glasses. After automatic focus, press the function button on the glasses or use voice control to shoot. It will automatically identify whether the object is abnormal or illegal, and display the top 3 most similar digits and credibility values ​​on the AR glasses.

[0012] The standard operation module guides port personnel to standardize the operation process according to the preset standard operation. The standard operation module presets different standard operation tasks, operation methods and operation steps in the background;

[0013] When the user starts a work task, the task name and work method will be displayed in the AR glasses. The user performs the operation according to the prompt information of the AR glasses, controls the AR glasses to execute the previous or next instruction through voice wake-up, performs voice-to-text input by calling keywords, and performs image capture through voice camera control. When performing standard operations, the entire operation is recorded through AR glasses; after the execution is completed, the entire operation process and the fully recorded video are automatically uploaded, and the standard operation process can be traced back through the background.

[0014] Furthermore, it also includes a remote collaboration module, which is a remote function auxiliary module with video, image and audio transmission functions. The front-line inspection personnel of the port can conduct one-to-one or one-to-many real-time voice and video exchanges with experts through video, image and audio transmission, remotely freeze the screen and perform remote drawing operations, and remotely collaborate to guide on-site personnel to carry out inspection work.

[0015] Furthermore, it also includes an intelligent identification module, which realizes the remote diagnosis and identification function. Through the image and voice modules of the AR glasses, difficult-to-inspect items during the on-site inspection are photographed and transmitted to the intelligent identification module. With the help of the artificial intelligence recognition function of the AI ​​module, or after the remote expert has identified through the identification module, the identification result is fed back to the AR glasses, realizing the combination of manual identification and artificial intelligence-assisted intelligent identification.

[0016] Furthermore, it also includes an auxiliary translation module, which obtains the surface text on the outer packaging box of the translated mail item through the shooting function of the AR glasses, and the image is sent to the translation auxiliary module for processing. The translation auxiliary module will automatically recognize the text language and translate it into Chinese, and automatically superimpose the translated text on the surface of the packaging box, and push the processed and superimposed image to the AR glasses for display.

[0017] Furthermore, the case module in the think tank search module includes national investigation cases and local inspection cases; front-line staff can use keyword fuzzy search catalogs and case contents for postal item inspection and comparison queries, which facilitates inspectors to quickly locate the attributes and legality of the inspected items.

[0018] Furthermore, in the AR recognition module, illegal publications are identified through barcodes on the publications; while endangered species, alien species, drugs, infringing counterfeits and gun parts are identified through AI image recognition.

[0019] Furthermore, the AR recognition module includes an AR glasses image input module, an image processing module, an AI image recognition module and an AI training module;

[0020] After the AR glasses take images through voice control, the image information is transmitted to the image processing module through the AR glasses image input module. After the image is pre-processed, it is transmitted to the AI ​​image recognition module for recognition, and then the recognition result is fed back to the AR glasses; and the AI ​​training module trains the AI ​​image recognition module to improve the efficiency and accuracy of image recognition.

[0021] Beneficial effects: The present invention has the following beneficial effects:

[0022] 1) Improve work efficiency: The AR glasses of the present invention can realize functions such as real-time visualization, data collection, and audio and video interaction, which improves the efficiency of customs supervision of postal items. The AR glasses of the present invention can be controlled by voice, which frees the hands and makes it more convenient for front-line staff to carry out parcel inspection;

[0023] 2) Improved accuracy: The AR glasses of the present invention use a deep convolutional neural network to conduct deep learning on the image feature data set to achieve the purpose of accurately identifying characteristic items. Then, through the collection and analysis of suspicious item graphic data, customs prohibited items can be identified more accurately. After superimposed translation software, various languages ​​can be quickly translated, and the accuracy of the inspection operation is greatly improved:

[0024] 3) Reduce office costs: The AR glasses of the present invention improve the credibility of contraband identification, save a lot of time cost of memorization and the manpower cost of consulting experts, and also reduce the use of paper documents, and realize the integration of six major functions, thus reducing office costs;

[0025] 4) Enhanced user experience: The use of the AR glasses of the present invention can provide users with a richer and more intuitive visual experience and interaction mode, allowing users to more conveniently master the operational skills of inspection and supervision, thereby improving user satisfaction and work convenience;

[0026] 5) Strengthening law enforcement support: The present invention sets up the establishment and continuous enrichment of the database and knowledge base of typical prohibited and restricted items of the customs in different categories for AI technology to learn and train. At the same time, it also realizes the integration and archiving backup of the supervision system such as customs technical regulations, typical cases, and inspection and issuance experience, which effectively promotes the long-term improvement of the customs law enforcement support and guarantee capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a structural diagram of the think tank retrieval module in the present invention;

[0028] Figure 2 This is a working diagram of the remote collaboration module in the present invention;

[0029] Figure 3 This is a flow chart of the standard operation module in the present invention;

[0030] Figure 4 A classification diagram of types recognized by the AR recognition module in the present invention;

[0031] Figure 5 This is the AR recognition module recognition flow chart of the present invention;

[0032] Figure 6 This is a workflow diagram of the smart identification module in the present invention;

[0033] Figure 7This is a workflow diagram of the auxiliary translation module in the present invention. DETAILED DESCRIPTION

[0034] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. The embodiments are implemented based on the technical solution of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0035] A wearable intelligent AR recognition glasses for customs, including a think tank retrieval module, an AR recognition module and a standard operation module;

[0036] like Figure 1 As shown in the figure, the think tank search module includes the prohibited and restricted catalog module and the case module. The prohibited and restricted catalog module includes the dangerous chemicals catalog, psychotropic drug catalog, narcotic drug catalog, CITES Convention, precursor chemicals catalog, prohibited animals and plants and products and other quarantine items, and other catalogs for customs inspection;

[0037] like Figure 4 As shown, the AR recognition module has a shooting function and a voice input control function, and the types of recognition of the AR recognition module include illegal publications, endangered species, alien species, drugs, infringement and counterfeiting, and gun parts;

[0038] Use the voice input function of the AR glasses to select a module. After entering the module, use the AR glasses to move the object to be identified to the front of the glasses. After automatic focus, press the function button on the glasses or use voice control to shoot. It will automatically identify whether the object is abnormal or illegal, and display the top 3 most similar digits and credibility values ​​on the AR glasses.

[0039] like Figure 3 As shown, the standard operation module guides port personnel to standardize the operation process according to the preset standard operation. The standard operation module presets different standard operation tasks, operation methods and operation steps in the background;

[0040] When the user starts a work task, the task name and work method will be displayed in the AR glasses. The user performs the operation according to the prompt information of the AR glasses, controls the AR glasses to execute the previous or next instruction through voice wake-up, performs voice-to-text input by calling keywords, and performs image capture through voice camera control. When performing standard operations, the entire operation is recorded through AR glasses; after the execution is completed, the entire operation process and the fully recorded video are automatically uploaded, and the standard operation process can be traced back through the background.

[0041] like Figure 2As shown, it also includes a remote collaboration module, which is a remote function auxiliary module with video, image and audio transmission functions. The front-line inspection personnel of the port can conduct one-to-one or one-to-many real-time voice and video exchanges with experts through video, image and audio transmission, remotely freeze the screen and perform remote drawing operations, and remotely collaborate to guide on-site personnel to implement inspection work.

[0042] like Figure 6 As shown, it also includes a smart identification module. The smart identification module realizes the remote diagnosis and identification function. Through the image and voice modules of the AR glasses, the difficult-to-inspect items during the on-site inspection are photographed and transmitted to the smart identification module. With the help of the artificial intelligence recognition function of the AI ​​module, or after the remote expert has identified through the identification module, the identification result is fed back to the AR glasses, realizing the combination of manual identification and artificial intelligence-assisted smart identification.

[0043] like Figure 7 As shown, it also includes an auxiliary translation module. The auxiliary translation module obtains the surface text on the outer packaging box of the translated mail item through the shooting function of the AR glasses. The image is sent to the translation auxiliary module for processing. The translation auxiliary module will automatically recognize the text language and translate it into Chinese, and automatically superimpose the translated text on the surface of the packaging box, and push the processed and superimposed image to the AR glasses for display.

[0044] The case module in the think tank search module includes national investigation cases and local inspection cases; front-line staff can use keyword fuzzy search catalogs and case contents for postal item inspection and comparison queries, which facilitates inspectors to quickly locate the attributes and legality of the items to be inspected.

[0045] In the AR recognition module, illegal publications are identified through the barcodes on the publications; endangered species, alien species, drugs, infringing counterfeits, and gun parts are identified through AI image recognition.

[0046] The AR recognition module includes an AR glasses image input module, an image processing module, an AI image recognition module, and an AI training module;

[0047] After the AR glasses take images through voice control, the image information is transmitted to the image processing module through the AR glasses image input module. After pre-processing the image, it is transmitted to the AI ​​image recognition module for recognition, and then the recognition result is fed back to the AR glasses; and the AI ​​training module trains the AI ​​image recognition module to improve the efficiency and accuracy of image recognition.

[0048] The standard operation function in the present invention takes the inspection of international mail supervision sites as an example:

[0049] In the first step, the AR glasses will use voice and screen text prompts to check whether the fence is missing or damaged, whether the passage door is closed in time, and ask the staff to take photos and record them;

[0050] In the second step, the AR glasses prompt the staff to check whether the various mails are stored in different categories and whether there are any mails that have not been declared to the customs for more than three months. The staff are asked to take photos and record them.

[0051] In the third step, the AR glasses prompt you to check whether the video surveillance is operating normally, whether there are exposed wires, air conditioning leaks, whether the fire safety facilities are in place, etc.

[0052] Users can set up and write standard operation function modules according to their actual work. Each standard operation process contains two levels. The first level is to set the operation steps and describe the key points of each step. The second level is to set whether to take photos or text records in each step. A standard operation process contains multiple operation processes. After the operation process is completed, the process record data and audio and video multimedia record files will be automatically uploaded. The main function of this function is to standardize daily work, eliminate the content of programmed mechanical memory, and achieve paperless work.

[0053] like Figure 5 As shown, the AR recognition module of the present invention has an intelligent recognition function. Relying on the physical image and the commodity barcode, the intelligent object recognition technology is established, focusing on solving the problem of rapid and accurate recognition of prohibited and restricted import and export items such as controlled drugs, illegal publications, endangered species, alien species, infringing items, and gun parts; specifically, the following steps are included:

[0054] 1) Establish a database of key prohibited and restricted items for entry and exit. For the six major categories of prohibited and restricted items, such as controlled drugs, for each prohibited and restricted item (such as triazolam among controlled drugs), the staff will collect no less than 50 high-definition feature pictures and eliminate possible interference information. The pictures of these prohibited and restricted items will form a basic database.

[0055] 2) Extract image features, use the latest computer software to extract the color, shape, texture and spatial features of the images in the database to generate a feature data set;

[0056] 3) Deep learning training: Use deep convolutional neural networks to conduct deep learning on feature data sets to achieve the purpose of accurately identifying feature items. First, classify and store common and typical prohibited and restricted items, use intelligent learning engines to extract and refine the differentiated features of different items, and perform serialized storage of these differentiated features to form a background comparison library;

[0057] 4) Intelligent image recognition: staff members use voice prompts to ask AR glasses to take pictures of suspected prohibited or restricted items. After comparison with the background library, the glasses will display the recognition results and credibility in the upper left corner of the lens screen, effectively improving the on-site inspection and early warning capabilities. The method of identifying illegal publications is to pre-import the ISBN international standard book number into the background to form a library, and scan the book barcode information through AR glasses. The background will automatically determine whether the book has been checked and issued and display the search information on the AR glasses.

[0058] The AR glasses of the present invention can realize functions such as real-time visualization, data collection, and audio and video interaction, which improves the efficiency of customs supervision of postal items. The AR glasses of the present invention can be controlled by voice, which frees the hands and makes it more convenient for front-line staff to carry out parcel inspection. The AR glasses of the present invention use deep convolutional neural networks to conduct deep learning of image feature data sets to achieve the purpose of accurately identifying characteristic items. Then, through the collection and analysis of suspicious item graphic data, customs prohibited items can be identified more accurately; after superimposing translation software, various languages ​​can be quickly translated, and the accuracy of inspection operations is greatly improved:

[0059] The AR glasses of the present invention improve the credibility of contraband identification, save a lot of time cost of memorization and manpower cost of consulting experts, and also reduce the use of paper documents, and realize the integration of six major functions to reduce office costs; the use of the AR glasses of the present invention can provide users with a richer and more intuitive visual experience and interaction method, so that users can more conveniently master the operational skills of inspection and supervision, and improve user satisfaction and work convenience.

[0060] The present invention sets up the establishment and continuous enrichment of the database and knowledge base of typical customs prohibited and restricted items in different categories for AI technology to learn and train. At the same time, it also realizes the integration and archiving backup of supervision systems such as customs technical regulations, typical cases, and inspection and issuance experience, which effectively promotes the substantial improvement of customs law enforcement support and guarantee capabilities.

[0061] The above specific implementation is only a preferred embodiment of the present invention and is not intended to limit the implementation of the present invention and the scope of the claims. All equivalent changes and modifications made according to the content of the patent protection scope of the present invention should be included in the scope of the patent application of the present invention.

Claims

1. A wearable smart AR recognition glasses for customs, characterized by: Including think tank retrieval module, AR recognition module and standard operation module; The think tank search module includes a prohibited and restricted catalog module and a case module. The prohibited and restricted catalog module includes a catalog of dangerous chemicals, a catalog of psychotropic drugs, a catalog of narcotic drugs, the CITES Convention, a catalog of precursor chemicals, a list of prohibited animals and plants and products and other quarantine items, and other catalogs for customs inspection; The AR recognition module has a shooting function and a voice input control function, and the types of recognition of the AR recognition module include illegal publications, endangered species, alien species, drugs, infringement and counterfeiting, and gun parts; Use the voice input function of the AR glasses to select a module. After entering the module, use the AR glasses to move the object to be identified to the front of the glasses. After automatic focus, press the function button on the glasses or use voice control to shoot. It will automatically identify whether the object is abnormal or illegal, and display the top 3 most similar digits and credibility values ​​on the AR glasses. The standard operation module guides port personnel to standardize the operation process according to the preset standard operation. The standard operation module presets different standard operation tasks, operation methods and operation steps in the background; When the user starts a work task, the task name and work method will be displayed in the AR glasses. The user performs the operation according to the prompt information of the AR glasses, controls the AR glasses to execute the previous or next instruction through voice wake-up, performs voice-to-text input by calling keywords, and performs image capture through voice camera control. When performing standard operations, the entire operation is recorded through AR glasses; after the execution is completed, the entire operation process and the fully recorded video are automatically uploaded, and the standard operation process can be traced back through the background.

2. The wearable smart AR recognition glasses for customs according to claim 1, characterized in that: It also includes a remote collaboration module, which is a remote function auxiliary module with video, image and audio transmission functions. The front-line inspection personnel at the port can conduct one-to-one or one-to-many real-time voice and video exchanges with experts through video, image and audio transmission, remotely freeze the screen and perform remote drawing operations, and remotely collaborate to guide on-site personnel to carry out inspection work.

3. The wearable smart AR recognition glasses for customs according to claim 1, characterized in that: It also includes an intelligent identification module, which realizes remote diagnosis and identification functions. Through the image and voice modules of the AR glasses, difficult-to-inspect items during the on-site inspection are photographed and transmitted to the intelligent identification module. With the help of the artificial intelligence recognition function of the AI ​​module, or after identification by remote experts through the identification module, the identification results are fed back to the AR glasses, realizing manual identification combined with artificial intelligence-assisted intelligent identification.

4. The wearable smart AR recognition glasses for customs according to claim 1, characterized in that: It also includes an auxiliary translation module, which obtains the surface text on the outer packaging box of the translated mail item through the shooting function of the AR glasses. The image is sent to the translation auxiliary module for processing. The translation auxiliary module will automatically recognize the text language and translate it into Chinese, and automatically superimpose the translated text on the surface of the packaging box, and push the processed and superimposed image to the AR glasses for display.

5. The wearable smart AR recognition glasses for customs according to claim 1, characterized in that: The case module in the think tank search module includes national investigation cases and local inspection cases; front-line staff can use keyword fuzzy search catalogs and case contents for postal item inspection and comparison queries, so that inspectors can quickly locate the attributes and legality of the items to be inspected.

6. The wearable smart AR recognition glasses for customs according to claim 1, characterized in that: In the AR recognition module, illegal publications are identified through the barcodes on the publications; while endangered species, alien species, drugs, infringing counterfeits and gun parts are identified through AI image recognition.

7. The wearable smart AR recognition glasses for customs according to claim 6, characterized in that: The AR recognition module includes an AR glasses image input module, an image processing module, an AI image recognition module and an AI training module; and the specific recognition operation steps are as follows: 1) Establish a database of key prohibited and restricted items for entry and exit: For six major categories of prohibited and restricted items such as controlled drugs, staff will collect no less than 50 high-definition feature images for each prohibited and restricted item, and remove any possible interference information. The images of these prohibited and restricted items will form a basic database; 2) Extract image features: Use the latest computer software to extract the color, shape, texture and spatial features of the images in the database to generate a feature data set; 3) Deep learning training: Use deep convolutional neural networks to conduct deep learning on feature data sets to achieve the purpose of accurately identifying feature objects; 4) Intelligent image recognition: The staff uses voice prompts to the AR glasses to take pictures of the characteristics of suspected prohibited or restricted items. After comparison with the background gallery, the glasses will display the recognition results and credibility in the upper left corner of the lens screen, effectively improving the on-site inspection and early warning capabilities. The method of identifying illegal publications is to pre-import the ISBN international standard book number and then take a picture of the corresponding book number for identification.

Images