An augmented reality-based automobile part detection method and device

Abstract

The application discloses a kind of based on augmented reality automobile parts detection method, applied to augmented reality server side, the method includes: S1: receiving the scanned real image information uploaded by augmented display client;S2: in the preset image recognition model dataset Filter out the image recognition sample associated with the real image information;S3: based on the image recognition sample of screening generates the image recognition model data subset corresponding to the real image information, based on the image recognition model data subset to scanned real image information carries out image recognition;S4: when image recognition succeeds, issue fault identification instruction and output fault identification result to the augmented reality client;S5: data processing is carried out to the real image in the real image information, constructs real image coordinate, constructs virtual image according to real image coordinate, and real image and the constructed virtual image are jointly output to the augmented reality client.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of equipment detection, in particular to an automobile part detection method and device based on augmented reality. BACKGROUND

[0002] In the process of stamping of automobile plates and assembly of automobiles, checking whether the stamped plates and parts meet the standard and whether there is damage on the appearance is indispensable in the stamping and part assembly link of an intelligent automobile factory. At present, the general detection method is still that a first group of workers manually check the specifications and appearance of parts with ordinary tools and naked eyes, and a second group of workers manually check and score, so that the detection efficiency is low. In addition, for the checking of small parts, workers are prone to fatigue when operating, and missed detection and false detection occur, which leads to some safety accidents. In addition, the detection of automobile parts needs to rely on workers with rich experience for judgment, and it is not easy for new workers to get started.

[0003] Therefore, it is of great significance to provide an automobile part detection method and device based on augmented reality in view of the above problems. SUMMARY

[0004] The purpose of the application is to provide an automobile part detection method and device based on augmented reality, which uses an image recognition algorithm of artificial intelligence in combination with a graphical visualization detection prompt of augmented reality to assist a detection worker in detecting the quality of parts and improve the detection efficiency of parts.

[0005] To solve the above technical problems, the application adopts the following scheme:

[0006] An automobile part detection method based on augmented reality is applied to an augmented reality server, and the method comprises the following steps:

[0007] S1: receiving scanned real object image information uploaded by an augmented reality client;

[0008] S2: screening an image recognition sample associated with the real object image information from a preset image recognition model data set;

[0009] S3: generating an image recognition model data subset corresponding to the real object image information based on the screened image recognition sample, and pushing the image recognition model data subset to the augmented reality client, so that the augmented reality client performs image scanning in a real-time environment, and performs image recognition on the scanned real object image information based on the image recognition model data subset;

[0010] S4: when the image recognition is successful, performing image recognition on the scanned real object image information based on the image recognition model data subset, and generating a corresponding image recognition result; and S5: pushing the image recognition result to the augmented reality client, so that the augmented reality client displays the image recognition result in a real-time environment.

[0011] The like information is subjected to fault recognition, and the fault recognition result is output to the augmented reality client;

[0012] S5: data processing is performed on the real object image information, real object image coordinates are constructed, a virtual image is constructed on the real object image according to the real object image coordinates by using a neural network algorithm, and the real object image and the constructed virtual image are jointly output to the augmented reality client;

[0013] S6: before S2 is executed, real object sample data is collected, defects in the real object sample are labeled, different defects are classified and corresponding labels are attached, AI deep learning training is performed on the real object sample data, an image recognition model data set is established, and after the training is completed, different defects defined in advance can be recognized;

[0014] S7: display data configured for the image recognition sample in the image recognition model data set is acquired, and a display effect configured for the display data is acquired, the acquired display data and the display effect are stored in association with the image recognition sample in the image recognition model data set, and in response to an update request of the augmented reality client, the display data and the display effect are dynamically updated;

[0015] S8: the image recognition algorithm can be based on image six-face recognition comparison and recognition. First, the product sample data of the parts is acquired, including the images of the product samples of various different defects and six faces of the product samples. Then, the types of defects of the customers and the detection requirements are collected, the classification method is planned, the area is manually labeled, the area is segmented according to the pixel level, the segmented area is merged to be the entire product for detection, then the area with complex shape can be drawn by using the automatic extraction and label making method, after the labeling is completed, the area is saved in the corresponding classification label, the image of the fault of different defects is subjected to enhancement processing, after the area, length, width, pixel ratio, pixel texture distribution and gray scale of the six faces are acquired, the above features are aggregated as a set, as a feature set vector, each feature is trained according to the feature set vector, the model parameters are determined, the corresponding defect image model is established according to the model parameters, the established image model is stored in the system database, at the same time, the same defect sample is superimposed and learned, and learning feedback training is performed until the detection accuracy of the feedback training reaches more than 99%, and in the learning process, other defect samples can also be added for AI learning at the same time.

[0016] Further, the method comprises:

[0017] When a part corresponding to any image recognition sample in the image recognition model data subset is identified from the real object image information, the display data associated with the image recognition sample and the display effect associated with the display data are pushed to the augmented reality client to enable the augmented reality client to enhance the display data in the augmented reality scene at the position corresponding to the part based on the display effect.

[0018] Further, the method comprises: when S5 is executed, acquiring a real-time image of a real object captured by a position sensor, and constructing a coordinate based on the real-time image of the real object.

[0019] Further, the method comprises: when S5 is executed, performing data processing on the real object image in the real object image information to identify, filter and mark the part to be detected and the background, so as to distinguish the part to be detected from the background.

[0020] Further, the data processing on the real object image in the real object image information further comprises virtual augmentation, background superposition, color filtering, image segmentation, binarization and edge detection, so as to effectively distinguish the part to be detected from the background.

[0021] Further, after the part to be detected is effectively distinguished from the background, a virtual image of a reference state of each part is constructed, and the real-time image of each part is continuously compared with the virtual image of the reference state.

[0022] The application further provides an automobile part detection device based on augmented reality, which is applied to an augmented reality server and comprises:

[0023] A receiving module receives the scanned real object image information uploaded by the augmented reality client;

[0024] A screening module screens an image recognition sample associated with the real object image information from a preset image recognition model data set;

[0025] A pushing module generates an image recognition model data subset corresponding to the real object image information based on the screened image recognition sample, and pushes the image recognition model data subset to the augmented reality client, so that the augmented reality client performs image recognition on the scanned real object image information based on the image recognition model data subset when performing image scanning in a real-time environment;

[0026] A first identification module receives the scanned real object image information uploaded by the augmented reality client, and performs image recognition on the scanned real object image information based on the image recognition model data subset;

[0027] a second identification module, when the image recognition is successful, issuing a fault identification instruction and outputting a fault identification result to the augmented reality client;

[0028] an image generation module, performing data processing on a real object image in the real object image information, constructing a real object image coordinate, constructing a virtual image on the real object image according to the real object image coordinate by using a neural network algorithm, and outputting the real object image and the constructed virtual image to the augmented reality client together.

[0029] Further, the application further comprises:

[0030] a training collection module, collecting real object sample data, labeling defects in the real object sample, classifying different defects and pasting corresponding labels, performing AI deep learning training on the real object sample data, establishing an image recognition model data set, and after the training is completed, being capable of identifying different defects of the pre-defined faults;

[0031] an acquisition module, acquiring display data configured for an image recognition sample in the image recognition model data set, and a display effect configured for the display data;

[0032] a storage module, associating and storing the acquired display data and the display effect with the image recognition sample in the image recognition model data set, and in response to an update request of the augmented reality client, dynamically updating the display data and the display effect.

[0033] The application has the beneficial effects that:

[0034] 1. The application uses the image recognition algorithm of artificial intelligence and the graphical visual detection prompt of augmented reality to assist the detection personnel in detecting the quality of the parts and components, improves the detection efficiency of the parts and components, is simple to operate, easy to use and promote, reduces the burden of workers, reduces the threshold requirement of the detection personnel, and greatly improves the daily inspection work efficiency of the basic maintenance and support.

[0035] 2. The application presents the visual problem icon in the form of augmented reality, can very intuitively reflect the problem reason and problem position of the automobile plate and parts and components, and provides a comparison reference figure, the area where the problem occurs is highlighted in the real object image, the trained identification algorithm for the automobile stamping plate and parts and components can also quickly and accurately identify the model and the problem, and the detection personnel can perform final confirmation and reporting according to the identification result, compared with the manual form, the identification speed and accuracy are improved. BRIEF DESCRIPTION OF DRAWINGS

[0036] Fig. 1 is a workflow diagram of the application;

[0037] Fig. 2 is an architecture diagram of the augmented reality server and the augmented reality client of the present application. DETAILED DESCRIPTION

[0038] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application. EMBODIMENT

[0039] As shown in Fig. 1 and Fig. 2, an augmented reality-based automobile part detection method is applied to an augmented reality server, and the method comprises:

[0040] S1: receiving scanned real object image information uploaded by an augmented reality client;

[0041] Specifically, the augmented reality client is installed on a terminal device such as a mobile phone or a tablet computer, and a camera on the device scans and collects real-time environment images, which include real object image information of the parts.

[0042] S2: screening out image recognition samples associated with the real object image information from a preset image recognition model data set;

[0043] Specifically, real object sample data is collected before S2 is executed, defects in the real object sample are labeled, different defects are classified and corresponding labels are attached, AI deep learning training is performed on the real object sample data, an image recognition model data set is established, and after the training is completed, different defects defined in advance can be recognized.

[0044] In an implementable embodiment, the image recognition algorithm can be based on image six-face recognition comparison recognition. First, product sample data of parts is obtained, including product samples of various defects and images of six faces of the product samples. Then, customer defect types and detection requirements are collected, a classification method is planned, and a region is manually labeled. The region is segmented by pixel level, and the segmented region is merged to be the entire product for detection. Then, the region with complex shape can be drawn by using automatic extraction and label making. After the labeling is completed, it is saved in the corresponding classification label. The fault images of different defects are enhanced. After the areas, lengths, widths, pixel ratios, pixel texture distributions, and gray scales of the six faces are obtained, the above features are aggregated into a set as a feature set vector. According to the feature set vector, each feature is trained to determine the model parameters. According to the model parameters, a corresponding defect image model is established. The established image model is stored in the system database, and the same defect samples are superimposed and learned, and learning feedback training is performed until the detection accuracy rate reaches more than 99%. In the learning process, other defect samples can also be added for AI learning at the same time.

[0045] In an implementable embodiment, the image recognition algorithm is also based on object combination recognition algorithm. First, a sample part image set is obtained, and the sample part image set is digitally processed to obtain a sample part image. Cluster analysis is performed according to the sample part image set and pose information to obtain a cluster image. Feature points are obtained by feature extraction on the sample part image. The feature points of different images are matched to obtain the homonymic points of the sample part image. Binding constraint calculation is performed according to the sample part image set, pose information, and the homonymic points to obtain a sparse point cloud. Multi-view stereo matching is performed according to the sparse point cloud and the cluster image to obtain a dense point cloud, and the dense point cloud is rendered to obtain a three-dimensional model of the sample part.

[0046] A cloud model library of sample parts is constructed according to the three-dimensional model of the sample parts. Image data of all parts to be recognized is collected and matched with the three-dimensional model of the sample parts in the cloud model library to recognize all parts to be recognized. A plurality of associated parts are selected from all parts to be recognized, the plurality of associated parts can be combined into one or more whole parts, and a three-dimensional model group corresponding to the plurality of associated parts is obtained. The three-dimensional model group corresponding to the plurality of associated parts is combined to obtain a three-dimensional combined model, and the three-dimensional combined model is used for recognition comparison with the whole part composed of the plurality of associated parts.

[0047] When the above algorithm model training is established, the docking test with the application terminal is performed, and after the test is successful, the augmented reality server is applied.

[0048] S3: generating an image recognition model data subset corresponding to the real object image information based on the screened image recognition samples, and pushing the image recognition model data subset to the augmented reality client, so that the augmented reality client performs image recognition on the scanned real object image information based on the image recognition model data subset when performing image scanning in a real-time environment;

[0049] In addition, the display data configured for the image recognition samples in the image recognition model data set is obtained, and the display effect configured for the display data is obtained; the obtained display data and the display effect are stored in association with the image recognition samples in the image recognition model data set, and the display data and the display effect are dynamically updated in response to an update request of the augmented reality client.

[0050] The display data and the display effect include but are not limited to part drawings, part models, problem icon prompts, part problem recognition, electronic rulers, matters needing attention queries, comparison reference maps, and part information statistics, etc.

[0051] When a part corresponding to any image recognition sample in the image recognition model data subset is recognized from the real object image information, the display data associated with the image recognition sample and the display effect associated with the display data are pushed to the augmented reality client, so that the augmented reality client enhances the display data on the position corresponding to the part in the augmented reality scene based on the display effect.

[0052] When the part is recognized, the recognized part drawing and the model of the recognized part can be obtained, and the drawing image of the part is mapped on the augmented service client and the electronic ruler function is provided to assist the inspector in comparing the size.

[0053] S4: when the image recognition is successful, performing fault recognition on the scanned real object image information based on the image recognition model data subset and outputting the fault recognition result to the augmented reality client;

[0054] The recognition result includes appearance problems such as automobile part image, paint peeling, damage, rust, etc., which are finally presented on the terminal screen of the augmented service client and can be marked by the inspector.

[0055] S5: performing data processing on the real object image in the real object image information, constructing a real object image coordinate, constructing a virtual image on the real object image according to the real object image coordinate by using a neural network algorithm, and outputting the real object image and the constructed virtual image to the augmented reality client.

[0056] Specifically, in the construction of the virtual image of the real object, the real-time image of the real object captured by the position sensor is obtained, and coordinates are constructed for the real-time image of the real object; the data processing of the real image information in the real object image is to identify, screen and identify the parts to be detected and the background, so as to distinguish the parts to be detected and the background; the data processing of the real image information in the real object image also includes virtual expansion, background superposition, color filtering, image segmentation, binarization and edge detection, so as to effectively distinguish the parts to be detected and the background; after the parts to be detected and the background are effectively distinguished, the virtual image of the reference state of each part is constructed, and the real image of each part is compared with the virtual image of the reference state in real time, so that the problems of the parts can be more directly observed through the comparison of the real image and the virtual image. Finally, a detection problem report is generated and uploaded.

[0057] The application further provides an automobile part detection device based on augmented reality, which is applied to an augmented reality server and comprises a receiving module, a screening module, a pushing module, a first identification module, a second identification module, an image generation module, a training and collecting module, an acquisition module and a storage module, and is used for implementing the automobile part detection method based on augmented reality.

[0058] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. According to the technical essence of the present application, any simple modification, equivalent replacement and improvement of the above embodiment within the spirit and principles of the present application are still within the protection scope of the technical scheme of the present application.

Claims

1. An augmented reality-based automobile component detection method, characterized by, The method is applied to an augmented reality server, and the method comprises the following steps: S1: receiving scanned real object image information uploaded by an augmented reality client; S2: screening out image recognition samples associated with the real object image information from a preset image recognition model data set; S3: generating an image recognition model data subset corresponding to the real object image information based on the screened out image recognition samples, and pushing the image recognition model data subset to the augmented reality client, so that the augmented reality client performs image recognition on scanned real object image information based on the image recognition model data subset when performing image scanning in a real-time environment; S4: when image recognition is successful, performing fault identification on the scanned real object image information based on the image recognition model data subset and outputting the fault identification result to the augmented reality client; S5: performing data processing on real object images in the real object image information, constructing real object image coordinates, constructing virtual images on the real object images by using a neural network algorithm according to the real object image coordinates, and jointly outputting the real object images and the constructed virtual images to the augmented reality client; S6: before S2, collecting real object sample data, labeling defects in the real object samples, classifying different defects and pasting corresponding labels, performing AI deep learning training on the real object sample data, establishing an image recognition model data set, and being capable of identifying faults of different defects defined in advance after training; S7: obtaining display data configured for image recognition samples in the image recognition model data set, and a display effect configured for the display data, storing the obtained display data and the display effect in association with the image recognition samples in the image recognition model data set, and dynamically updating the display data and the display effect in response to an update request of the augmented reality client; S8: the image recognition algorithm can be based on image six-face recognition and comparison. First, obtain product sample data of parts, including product samples of various different defects and images of six faces of the product samples. Then, collect customer defect types and detection requirements, plan a classification method, manually label regions, divide the regions by pixel level, merge the divided regions into the entire product for detection, then draw the regions with complex shapes by using an automatic extraction and label making method, save the labeled regions into corresponding classification labels after labeling, perform enhancement processing on fault images of different defects, obtain the area, length, width, pixel ratio, pixel texture distribution and gray scale of the six faces, aggregate the above features into a set as a feature set vector, train each feature according to the feature set vector, determine model parameters, establish a corresponding defect image model according to the model parameters, store the established image model in a system database, simultaneously perform superimposed learning on samples of the same defect, and perform learning feedback training until the detection accuracy of the feedback training reaches 99% or above. In the learning process, other defect samples can also be added for AI learning at the same time.

2. The method of claim 1, wherein the method further comprises: The method comprises: When a part corresponding to any image recognition sample in the image recognition model data subset is identified from the real object image information, the display data associated with the image recognition sample and the display effect associated with the display data are pushed to the augmented reality client to enable the augmented reality client to enhance the display data in the augmented reality scene at a position corresponding to the part based on the display effect.

3. The method of claim 1, wherein the method further comprises: The method comprises: when S5 is performed, acquiring real-time images of real objects captured by a position sensor, and constructing coordinates of the real-time images of real objects.

4. The method of claim 1, wherein the method further comprises: The method comprises: when S5 is performed, data processing of the real object images in the real object image information is performed to identify, filter and mark the parts to be detected and the background, so as to distinguish the parts to be detected from the background.

5. The method of claim 4, wherein the method further comprises: The data processing of the real object images in the real object image information further comprises virtual expansion, background superposition, color filtering, image segmentation, binarization and edge detection, so as to effectively distinguish the parts to be detected from the background.

6. The method of claim 5, wherein the method further comprises: After the parts to be detected are effectively distinguished from the background, virtual images of the parts in a reference state are constructed, and real-time comparison between the real object images of the parts and the virtual images of the parts in the reference state is continuously performed.

7. An augmented reality-based automobile component inspection apparatus for implementing the augmented reality-based automobile component inspection method according to claim 1, characterized by, The device comprises: a receiving module configured to receive real object image information uploaded by an augmented reality client; a screening module configured to screen, from a preset image recognition model data set, an image recognition sample associated with the real object image information; a pushing module configured to generate an image recognition model data subset corresponding to the real object image information based on the screened image recognition sample, and push the image recognition model data subset to the augmented reality client, so that the augmented reality client performs image recognition on the scanned real object image information based on the image recognition model data subset when performing image scanning in a real-time environment; a first identification module configured to receive real object image information uploaded by an augmented reality client, and perform image recognition on the scanned real object image information based on the image recognition model data subset; a second identification module configured to issue a fault identification instruction and output a fault identification result to the augmented reality client when the image recognition is successful; an image generation module configured to perform data processing on real object images in the real object image information, construct coordinates of the real object images, construct virtual images on the real object images based on the coordinates of the real object images using a neural network algorithm, and output the real object images and the constructed virtual images to the augmented reality client.

8. The augmented reality based automobile component detection device as claimed in claim 7, wherein, Further comprising: a training and collecting module configured to collect real object sample data, mark defects in the real object sample, classify different defects and attach corresponding labels, perform AI deep learning training on the real object sample data, establish an image recognition model data set, and identify different defects after the training is completed; an acquisition module configured to acquire display data configured for image recognition samples in the image recognition model data set; and a display effect configured for the display data. The storage module stores the obtained display data and the display effect in association with image recognition samples in the image recognition model data set, and dynamically updates the display data and the display effect in response to an update request of the augmented reality client.

Images