An image recognition-based aviation assembly manufacturing process data checking method and device

Abstract

The present application relates to the field of aircraft manufacturing production management and control, and particularly relates to an aviation assembly manufacturing process data verification method and device based on image recognition. The method comprises the following steps: firstly, binding the position information and the camera information to obtain the video data of the production site; secondly, pulling the streaming media data from the video data, intercepting the real-time position map according to a fixed period, and marking the real-time position map; further, pre-processing and calculating the pre-acquired empty position map and the real-time position map to determine the position state; finally, binding the real-time position map with the corresponding position information in the MES, obtaining the aircraft physical flow information after processing, and comparing and fusing the physical flow information with the information flow data in the MES. The present application can accurately determine the working time of the aircraft at each position, realize the consistency of the physical flow and the information flow, and provide strong support for the precise management and control of the whole life cycle of the aircraft manufacturing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of aircraft manufacturing production management, and particularly relates to an aviation assembly manufacturing process data verification method and device based on image recognition. BACKGROUND

[0002] Airplane assembly is an important stage for realizing the function of the airplane and a key means for forming the quality characteristics of the airplane, and is an important link of airplane manufacturing, and has a long production cycle, covers a wide range of technologies, and has been focused on by production managers. In recent years, with the in-depth promotion of lean production and the gradual application of pulsating production mode, higher requirements are put forward for the fine management of the airplane production process. At present, all-around management has been basically realized in the airplane assembly workshop, and airplane assembly MES has been widely applied in major manufacturers, and to a certain extent, fine management of the assembly production site has been realized, but due to the use of manual feedback information in the MES, there are a large number of inconsistencies between the information flow and the physical flow in the existing system. SUMMARY

[0003] The present application aims at the inconsistency between the physical flow and the information flow in the airplane assembly process in the prior art, and provides an aviation assembly manufacturing process data verification method and device based on image recognition, real-time information of a production site is collected by using an existing monitoring camera in the production site, a camera in an assembly site is bound with an assembly position, an image recognition technology is used to process the collected images on site, it is judged whether an airplane is working at the current position, and production data in the MES are fused, so that the working time of the airplane at each position can be accurately determined, the consistency of the physical flow and the information flow is realized, and strong support is provided for the precise management of the whole life cycle of the airplane manufacturing.

[0004] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0005] An aviation assembly manufacturing process data verification method based on image recognition comprises the following steps:

[0006] S1, binding position information and camera information, obtaining video data of a production site;

[0007] S2, pulling streaming media data from the video data, intercepting real-time position images according to a fixed period, and marking the real-time position images;

[0008] S3, preprocessing and calculating the pre-acquired empty position images and the real-time position images to judge the position state;

[0009] S4, binding the real-time position images with the corresponding position information in the MES, obtaining airplane physical flow information after processing, and comparing and fusing the physical flow information with information flow data in the MES.

[0010] As a preferred scheme of the present application, in step S1, a unique corresponding camera is selected for each station to bind information, and the station information includes the aircraft, the jig, and related facilities and equipment.

[0011] As a preferred scheme of the present application, in step S2, based on the ESDK service of the monitoring video system, the camera device number and NVR code are taken as inputs to request the ESDK service, the ESDK service returns a real-time video stream encryption address, the RTSP video stream is parsed and converted into an FLV protocol video stream, based on the FLV protocol video stream, a fixed period t for intercepting a real-time station picture is set, and a timestamp is marked on the real-time station picture.

[0012] As a preferred scheme of the present application, in step S3, an image binarization method is used to pre-process the empty station picture and the real-time station picture.

[0013] As a preferred scheme of the present application, in step S3, calculation includes difference operation on the empty station picture and the real-time station picture, and based on the feature extraction mode of the sobel operator, image edge features are extracted to determine the state, and the formula is as follows:

[0014] The image G after transverse edge detection is x :

[0015]

[0016] The image G after longitudinal edge detection is y :

[0017]

[0018] The gradient direction angle is

[0019] θ = arctan(G y / G x )

[0020] wherein P is the image after difference operation on the empty station picture and the real-time station picture; if the gradient direction angle θ is equal to zero, the real-time station picture has an aircraft; otherwise, the real-time station picture does not have an aircraft.

[0021] As a preferred scheme of the present application, in step S4, the real-time station picture is bound with the corresponding station information in the MES, a station state table is constructed, the state table is converted into a state sequence, aircraft physical flow information is formed, and the aircraft is moved to the next station A mThe actual logistics information for boarding an aircraft can be represented as follows:

[0022]

[0023] in, Indicates in A m The k-th time the start time of aircraft operation is detected at the parking position. Indicates in A m From the camera position From the start to the end when no aircraft is detected, For the aircraft in A m The actual operation time of the kth operation at the machine position.

[0024] As a preferred embodiment of the present invention, a data verification method for aerospace assembly and manufacturing processes based on image recognition, wherein in step S4, the data at stand A is obtained from the MES. m The information flow data on the screen can be represented as:

[0025]

[0026] in, Indicates in A m The start time of the k-th aircraft operation at the parking position. Indicates in A m The end time of the k-th aircraft operation at the parking position. For the aircraft in A m The operation time of the kth operation at the machine position.

[0027] As a preferred embodiment of the present invention, a data verification method for aerospace assembly manufacturing process based on image recognition, wherein step S4 compares and merges the physical flow information with the information flow data in the MES, including the following steps:

[0028] S41, from aircraft logistics information Extraction position A m The start time is obtained from the time of the k-th operation segment. and end time

[0029] S42, based on the start time Information flow data Search and find the distance The most recent start time is denoted as Its corresponding end time is denoted as

[0030] S43, Calculate the overlap rate between two time periods. When δ is greater than or equal to the threshold, the information flow data is determined. The work time should be revised to When the delta is less than the threshold, return to step S42, find the second closest start time and calculate delta, until the delta is greater than or equal to the threshold.

[0031] As a preferred scheme of the present application, an image recognition-based aviation assembly manufacturing process data verification device comprises a processing unit and a camera in communication connection with the processing unit, and the processing unit is used for processing and storing aviation assembly manufacturing process data verification workflows, so that the device can execute the above technical solutions.

[0032] In summary, due to the adoption of the above technical solutions, the present application has the following beneficial effects:

[0033] Based on video stream media data processing and real-time site map, the present application uses image recognition technology to process the empty site map and the real-time site map, judges whether the current site has an aircraft working, obtains aircraft physical flow information, accurately determines the working time of the aircraft at each site by combining the aircraft physical flow information and the information flow data in the MES, realizes the consistency of the physical flow and the information flow, and provides strong support for precise management and control of the whole life cycle of the aircraft. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a method flowchart of the present application. DETAILED DESCRIPTION

[0035] The present application will be described in detail below with reference to the accompanying drawings.

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0037] Example 1

[0038] An image recognition-based aviation assembly manufacturing process data verification method, characterized in that it comprises the following steps:

[0039] S1, bind the site information and the camera information to obtain the video data of the production site;

[0040] Specifically, a unique corresponding camera is selected for each site in the workshop to shoot video data of the production site, and the camera can cover all information of the site, including the aircraft, the jig, related facilities and equipment, etc. The camera and the site information are bound to form a corresponding table as shown in Table 1.

[0041] Table 1 camera site correspondence

[0042] Camera C03 C12 C15 C18 C20 C22 … C40 … Camera position A1 A2 A3 A4 A5 A6 … Am …

[0043] S2, pulling streaming media data from the video data, intercepting real-time camera position map in fixed period and marking it;

[0044] Specifically, based on the production site video data shot by the camera, the ESDK service of the monitoring video system is used, the production monitoring video cluster service master node IP to be accessed and the username and password for accessing the production monitoring system platform are configured in the service, the ESDK service registration function is completed, the camera device number and NVR code are taken as inputs, and the ESDK service is requested, based on RTSP protocol encapsulation, the ESDK service returns the real-time video stream encryption address of the production monitoring video, the RTSP video stream is parsed, and it is forwarded to the streaming media service through the rtmp protocol, and it is uniformly converted into FLV protocol video stream.

[0045] Further, according to the current management requirements of aircraft assembly, the interception picture period is set as t, the real-time camera position map of the position is obtained, and the real-time camera position map is marked with a time stamp for subsequent processing.

[0046] S3, preprocessing the empty camera position map and the real-time camera position map, performing difference operation on the empty camera position map and the real-time camera position map, and extracting image edge features based on the sobel operator feature extraction method to determine the state;

[0047] Specifically, the camera shoots a photo of the corresponding position without aircraft operation as the empty camera position map; in order to better obtain the image features, the image binarization method is used to process the empty camera position map and the real-time camera position map, specifically, since the environment in the workshop is relatively stable and the video shooting effect is ideal, according to the actual situation of the assembly workshop, after processing the empty camera position map and the real-time camera position map by using the global threshold method, corrosion and expansion operation are performed, which realizes the removal of noise in the real-time camera position map while retaining the effective information in the real-time camera position map;

[0048] Further, difference operation is performed on the preprocessed empty camera position map and the real-time camera position map to obtain the position of the aircraft in the image, and the region of the aircraft in the image is accurately marked;

[0049] Further, the sobel operator based feature extraction method is used to extract the image edge features to determine whether there is an aircraft in the image, the sobel operator contains two 3x3 matrices, which are horizontal and vertical, and plane convolution is performed on the image, and the horizontal and vertical brightness difference approximation values are obtained. Here, P represents the image obtained by subtracting the empty camera position map from the real-time camera position map, G x represents the image detected by the horizontal edge, and G y represents the image detected by the vertical edge, and the related algorithm is as follows:

[0050]

[0051] The transverse and longitudinal gradient approximation values of each pixel of the image are combined to obtain the gradient size of the pixel using the following manner:

[0052]

[0053] The gradient direction calculation method is as follows:

[0054] θ=arctan(G y / G x ) (4)

[0055] If the gradient direction angle θ is equal to zero, it means that the image has a longitudinal edge at this position, and the left side is darker than the right side. At this time, the real-time aircraft position map has an aircraft, otherwise, there is no aircraft.

[0056] S4, binding the real-time aircraft position map with the corresponding aircraft position information in the MES, obtaining the aircraft physical flow information after processing, and comparing and fusing the physical flow information with the information flow data in the MES.

[0057] After obtaining the judgment result of the real-time aircraft position map, the data is structured and processed, and is bound with the corresponding aircraft position information to construct an aircraft position state table, wherein T is the starting time, 1 represents that there is an aircraft in the real-time aircraft position map, and 0 represents that there is no aircraft in the real-time aircraft position map, and the aircraft position state table as shown in Table 2 is obtained.

[0058] Table 2 Aircraft position state table

[0059] T T+t T+2t T+3t T+4t T+5t T+6t … T+nt [A1] 0 0 1 1 1 1 0 0 [A2] 1 0 0 0 0 0 1 1 [A3] 0 0 0 0 0 0 0 0 … … … … … … … … … … A m ]]> 0 0 0 0 1 1 1 1

[0060] The state table is converted into a state sequence to form the aircraft physical flow information:

[0061]

[0062] Wherein, represents the time when the aircraft is first detected on the A m position, represents the time from the start to the time when the aircraft is not detected on the A position, is the actual working time of the aircraft on the A m position for the kth time; represents the start time of detecting the aircraft working on the A m position for the kth time, represents the end time of detecting the aircraft on the A m position from the start to the time when the aircraft is not detected, is the actual working time of the aircraft on the A m position for the kth time.The actual working time of the kth time at the position;

[0063] The data in the assembled MES is processed to obtain the aircraft working time of each position in the system, to form the current information flow data, which can be expressed as:

[0064]

[0065] Among them, represents the start time of the kth aircraft operation at A m position, represents the end time of the kth aircraft operation at A m position, is the working time of the kth time at A m position;

[0066] Taking A1 position as an example:

[0067]

[0068] Among them, s k represents the start time of the kth aircraft operation at A1 position, and e k is the end time;

[0069] At present, the working time of the aircraft at any position is greater than 24h, the physical flow information is compared and fused with the information flow data in the MES, and the fusion rule is as follows:

[0070] S41, taking the physical flow information as the benchmark, extracting the first working time of the position from the , obtaining the start time and the end time

[0071] S42, taking time as the benchmark, searching the information flow data , finding the nearest start time from time, recorded as , and the corresponding end time is recorded as

[0072] S43, calculate the coincidence rate of the two time periods:

[0073]

[0074] When δ is greater than or equal to the threshold value, it is determined that the actual start and end time of the information flow should be revised to When δ is less than the threshold value, step S42 needs to be re-executed to find the second nearest start time, obtain the time period, and recalculate δ until the time interval that meets the condition is found.

[0075] An image recognition-based aviation assembly manufacturing process data verification device comprises a processing unit and a camera in communication connection with the processing unit, and the processing unit is used for processing and storing aviation assembly manufacturing process data verification workflow.

[0076] In summary, after the accurate physical flow information is fused with the manufacturing process information in the MES, accurate manufacturing process data can be obtained, the time of working at each station in the aircraft assembly process can be accurately restored, the information flow of the aircraft manufacturing process is consistent with the physical flow, and strong support is provided for the refinement of production management.

[0077] The above merely describes preferred embodiments of the present application and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image recognition based method for verifying manufacturing process data for an aircraft assembly, the method comprising: The method comprises the following steps: S1, binding site information and camera information to obtain video data of a production site; S2, pulling streaming media data from the video data, intercepting a real-time site map at a fixed period, and marking the real-time site map; S3, preprocessing and calculating a pre-acquired empty site map and the real-time site map to determine a site state; S4, binding the real-time site map with corresponding site information in MES, obtaining aircraft physical flow information after processing, and comparing and fusing the physical flow information with information flow data in MES; In step S4, comparing and fusing the physical flow information with the information flow data in MES comprises the following steps: S41, extracting the time of the kth segment of the aircraft physical flow information from the aircraft physical flow information m to obtain the start time and the end time S42, based on the start time performing search on the information flow data the start time closest to the time, denoted as ​​ S43, calculate the coincidence rate of two time periods When δ is greater than or equal to the threshold value, then it is determined that the information flow data should be revised to When δ is less than the threshold value, return to step S42, find the second closest start time and calculate δ, until δ is greater than or equal to the threshold value.

2. The image recognition based data verification method for aircraft assembly manufacturing process according to claim 1, wherein, In step S1, each site selects a unique corresponding camera for information binding, and the site information includes an aircraft, a jig, and related facilities and equipment.

3. The image recognition based data verification method for aircraft assembly manufacturing process according to claim 1, wherein, In step S2, based on the ESDK service of a monitoring video system, the camera device number and NVR code are taken as inputs, the ESDK service is requested, the ESDK service returns a real-time video stream encryption address, the RTSP video stream is parsed and converted into an FLV protocol video stream, based on the FLV protocol video stream, a fixed period t for intercepting the real-time site map is set, and the real-time site map is marked with a timestamp.

4. The image recognition based data verification method for aircraft assembly manufacturing process according to claim 1, wherein, In step S3, the empty site map and the real-time site map are preprocessed by using an image binarization method.

5. The image recognition based data verification method for aircraft assembly manufacturing process according to claim 1, wherein, In step S3, the calculation comprises performing a difference operation on the empty site map and the real-time site map, and then extracting image edge features to determine the site state based on a sobel operator feature extraction method, and the formula is as follows: Image G transversely edge-detected x is: Image G detected by longitudinal edge y is: The gradient direction angle is: θ = arctan(G y / G x ) wherein P is an image obtained by performing a difference operation on the empty site map and the real-time site map; if the gradient direction angle θ is equal to zero, the real-time site map has an aircraft; otherwise, the real-time site map does not have an aircraft.

6. The image recognition based data verification method for aviation assembly manufacturing process according to claim 1, wherein, In step S4, the real-time seat map is bound with the corresponding seat information in the MES to construct a seat status table, the status table is converted into a status sequence, and the aircraft physical flow information is formed. The aircraft physical flow information at the seat A m The aircraft physical flow information can be expressed as wherein, represents the start time of the detection of the start of the aircraft operation at the A m position, represents the end time of the detection of the end of the aircraft at the A m position from the start time, is the actual operation time of the aircraft at the A m position for the kth time.

7. The image recognition based data verification method for aviation assembly manufacturing process according to claim 1, wherein, In step S4, retrieve the data from station A in the MES. m The information flow data on the screen can be represented as: wherein, denotes the start time of the kth aircraft operation at the A m gate, denotes the end time of the kth aircraft operation at the A m gate, is the kth operation time of the aircraft at the A m gate.

8. An image recognition based aviation assembly manufacturing process data verification apparatus, characterized in that, The device comprises a processing unit and a camera in communication connection with the processing unit, the processing unit is used for processing and storing an aviation assembly manufacturing process data verification work flow, so that the device can execute the method in any one of claims 1 to 7.

Images