Part target recognition method fusing prior structural knowledge

Abstract

The present application belongs to the technical field of computer vision, and particularly relates to a component target recognition method fusing prior structure knowledge, comprising: collecting inspection images of each region on complex instruments in a workshop, determining a standard image pre-collected by using position constraint mapping based on the collection position of each inspection image, and obtaining a multi-channel prior semantic image of the corresponding region obtained from the standard image; adopting a pre-constructed multi-channel semantic image input branch to fuse the last two layers of high semantic feature images generated by a trained inspection image feature extraction network for each inspection image and the multi-channel prior semantic image corresponding to the inspection image, so as to realize the enhancement of the prior structure knowledge on the inspection image features; wherein the multi-channel semantic image input branch is constructed by combining a Bottleneck module of Resnet and a deformable convolution operator; based on the fused feature images, the component target recognition result is obtained through regression and classification, so as to improve the component recognition precision.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and more particularly relates to a component target recognition method fusing prior structural knowledge. BACKGROUND

[0002] Component recognition on complex equipment in the workshop is the basis for complex equipment defect diagnosis. Among all types of defects, except for foreign matter, others are closely related to the components themselves. Therefore, the more accurate the component recognition result is, the better the defect diagnosis effect is. However, the current general target detection network has low precision when applied to complex equipment component detection, which cannot meet the demand. It is a feasible method to fuse prior structural knowledge into target detection to improve precision, but the shooting position and angle of the inspection image are variable, and the presented component prior structural knowledge is complex. The generalization performance is poor when the component prior structural knowledge is fixed in the GRU weight. Therefore, the method of learning structural knowledge into the network weight based on GRU and other recurrent neural network units is not applicable. The complex equipment in the workshop refers to the industrial production equipment system with precise and complex structure, such as rail transit vehicle bogie and power substation equipment.

[0003] Therefore, the existing component target recognition method has the problems of complex component structure, low component detection precision, and difficulty in fusing prior structural knowledge into target detection. SUMMARY

[0004] In view of the defects and improvement needs of the prior art, the present application provides a component target recognition method fusing prior structural knowledge, which aims to improve the recognition accuracy of components on complex equipment in the workshop.

[0005] To achieve the above-mentioned purpose, according to one aspect of the present application, a component target recognition method fusing prior structural knowledge is provided, comprising:

[0006] Collecting inspection images of each region on the complex equipment in the workshop, based on the collection position of each inspection image, determining a standard image pre-collected by using position constraint mapping, and obtaining a multi-channel prior semantic image of the corresponding region obtained from the standard image, wherein the multi-channel prior semantic image is constructed in the following manner: performing information conversion on the component annotation information of the standard image to obtain an information form capable of expressing the prior probability of the component on the image; initializing a multi-channel gray image with the number of channels consistent with the total number of all component types in the complex equipment; superimposing the information form on the multi-channel gray image to generate a multi-channel prior semantic image as the tensorized prior structural knowledge, and the prior semantic image of each channel represents the prior position and size of a certain component type in the standard image in the form of pixel value representing probability;

[0007] The pre-constructed multi-channel semantic image input branch is used to fuse the last two layers of high semantic feature images generated by the trained inspection image feature extraction network and the multi-channel prior semantic image corresponding to the inspection image, so as to enhance the prior structural knowledge to the inspection image features; wherein the multi-channel semantic image input branch is constructed by combining the bottleneck module of Resnet and the deformable convolution operator;

[0008] Based on the fused feature image and the feature image output by other output layers of the inspection image feature extraction network, the component target recognition result is obtained through regression and classification.

[0009] Further, the information conversion mode is Gaussian probability ellipse semantic information extraction.

[0010] Further, the inspection robot is used to collect the inspection images of each region on the complex machine in the workshop, and the complex machine in the workshop is inspected and photographed based on the 2D or 3D camera at the end of the inspection robot.

[0011] Further, the pixel value of the multi-channel gray image is a preset value, and the value range of the pixel value is [0, 1], and the length-width down-sampling of the multi-channel gray image is one-eighth of the standard image.

[0012] Further, the components on the complex machine in the workshop are divided into fixed components and rotating mobile components according to whether their positions are fixed, and different prior semantic image construction methods are used to construct the prior semantic image of the corresponding category channel.

[0013] Further, the prior semantic image of the corresponding category channel of each fixed component in the standard image is constructed in the following way:

[0014] Traverse each fixed component in the standard image, and generate a Gaussian ellipse two-dimensional semantic normal distribution of the fixed component according to its bounding box.

[0015] Add the Gaussian ellipse two-dimensional semantic normal distribution of each fixed component to the corresponding position of the gray image of the corresponding category channel, and if the Gaussian ellipse two-dimensional semantic normal distributions of the same type of components overlap, the relative high value of each pixel in the overlapping area is taken.

[0016] Further, the prior semantic image of the corresponding category channel of each mobile component in the standard image is constructed in the following way:

[0017] According to the motion trajectory of each type of component on the complex instrument, the type of curve and the formula of the trajectory projected on the standard image are determined; each moving component of the type in the standard image is traversed, the center of the bounding box of each moving component of the type is taken as a fitting point, and the specific formula parameters are fitted according to the type of curve and the formula to obtain a specific required curve formula;

[0018] The coordinates of the pole point of the required curve formula are calculated to obtain the x and y ranges of the curve in the image, wherein if the tangent line passing through a point on the curve is parallel to the x-axis or the y-axis, the point is referred to as the pole point of the curve in the x-axis or y-axis direction;

[0019] According to the x and y ranges of each curve in the image, a point set on the curve is calculated, and the points in the point set that are shielded in the image by the moving trajectory of the moving component of the corresponding type in the standard image are deleted;

[0020] A one-dimensional normal distribution, i.e., a Gaussian normal circle, is produced in the vicinity of each curve with the curve point set as the center, as the semantic distribution of the moving component of the corresponding type;

[0021] The semantic distribution of each type of rotating moving component is added to the corresponding position of the gray image of the corresponding category channel.

[0022] Further, the implementation manner of the multi-channel semantic image input branch for the fusion is as follows:

[0023] S1, the resolution of the multi-channel semantic image is reduced, then it is upgraded in dimension through a 1x1 convolution, and then it is down-sampled, so that the resolution of the multi-channel semantic image is consistent with that of the high semantic feature image output by the second last level of the inspection image feature extraction network;

[0024] S2, the multi-channel semantic image obtained in step S1 is connected with the high semantic feature image output by the second last level of the inspection image feature extraction network in the channel to obtain a tensor;

[0025] S3, the tensor is subjected to a deformable convolution operator to realize the preliminary alignment and fusion of the multi-channel semantic image and the feature image, and then K Bottleneck modules are used for feature extraction to obtain a first image feature fused with prior semantics;

[0026] S4, the first image feature fused with prior semantics is fused with the high semantic feature image output by the last level of the inspection image feature extraction network after maximum pooling; then T Bottleneck modules are used for feature extraction, and then a 1x1 convolution is performed to obtain a second image feature;

[0027] S5, the first image feature and the second image feature replace the last two level outputs of the original inspection image feature extraction network, and are input to the regression and separation network to realize the input and fusion of the prior semantic image in the feature extraction, wherein M, N, K and T are all positive integers.

[0028] The application further provides a component target recognition model construction method, comprising:

[0029] The component target recognition network is built, comprising an inspection image feature extraction network, a multi-channel semantic image input branch and a regression and separation network; a training sample set is constructed, the component target recognition network is trained, and a component target recognition model integrated with prior structural knowledge is obtained, which is used to execute the component target recognition method integrated with prior structural knowledge as described above.

[0030] The inspection image feature extraction network is used to extract a feature image of an inspection image, and the last two layers of the inspection image feature extraction network output a high semantic feature image.

[0031] The multi-channel semantic image input branch is used to fuse the last two layers of high semantic feature images generated by the inspection image feature extraction network for each inspection image and the multi-channel prior semantic image corresponding to the inspection image, so as to realize the enhancement of the prior structural knowledge on the inspection image feature; wherein the multi-channel semantic image input branch is constructed by combining a bottleneck module of Resnet and a deformable convolution operator.

[0032] The regression and classification network is used to obtain a component target recognition result based on the fused feature image and the feature images output by other output layers of the inspection image feature extraction network.

[0033] The application further provides a computer readable storage medium comprising a stored computer program, wherein the computer program controls a device where the storage medium is located to execute the component target recognition method integrated with prior structural knowledge and / or the component target recognition model construction method as described above when the computer program is run by a processor.

[0034] Overall, the above technical solutions conceived by the application can achieve the following beneficial effects:

[0035] (1) The application proposes a component target recognition method fusing prior structure knowledge, first, collecting inspection images at different positions, determining the corresponding pre-acquired standard image of each inspection image based on position constraints, to determine the corresponding prior semantic structure knowledge of each inspection image under the acquisition position and angle of view. Subsequently, the application uses an inspection image feature extraction network to extract features of each inspection image, and introduces a multi-channel semantic image input branch to adjust the last two layers of high semantic feature images obtained by the inspection image feature extraction network, and then inputs the two layers of high semantic feature images into the subsequent regression and classification network after fusing the prior semantic structure knowledge into the two layers of high semantic feature images, so as to enhance the prior structure knowledge to the inspection image features, and effectively improve the component recognition accuracy.

[0036] (2) The component labeling information of the standard image uses Gaussian probability ellipse semantic information extraction, which can enhance the original feature effect and improve the fuzzy boundary of detection.

[0037] (3) The components on the bogie can be divided into fixed components and rotating mobile components according to whether the position is fixed, and the two kinds of components use inconsistent semantic image construction methods, the constructed fixed component and rotating mobile component semantic images are merged to form tensorized prior structure knowledge, and this prior structure knowledge construction method can well represent the semantics (including type, position and length-width information) of the target component in each standard image, and can well adapt to the needs of the subsequent network. In this way, the accuracy of complex component semantic image construction can be improved, and the semantics, position and length-width information of the target in the standard image can be better represented.

[0038] (4) The neural network input module of the target detection framework is constructed by using the bottleneck module of Resnet and the deformable convolution operator, the last two layers of high semantic feature images generated by the inspection image feature extraction network and the prior semantic image are fused, the prior structure knowledge is enhanced to the inspection image features, and the component recognition accuracy is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A flowchart of a component target recognition method fusing prior structure knowledge provided by an embodiment of the application is shown in the figure;

[0040] Figure 2 A result diagram of a semantic image construction method for fixed components and rotating mobile components provided by another embodiment of the application is shown in the figure;

[0041] Figure 3 A structure diagram of a multi-channel semantic image input branch of Faster R-CNN FPN provided by an embodiment of the application is shown in the figure;

[0042] Figure 4A structural schematic diagram of a target recognition neural network input module of a YOLO v7 component is provided for another embodiment of the application. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0044] Embodiment one

[0045] A component target recognition method fusing prior structure knowledge, as shown in Figure 1 , comprises:

[0046] S1, collecting inspection images of each area on complex equipment in a workshop, based on the collection position of each inspection image, determining a standard image pre-collected by using position constraint mapping, obtaining a multi-channel prior semantic image of the corresponding area obtained from the standard image, wherein the multi-channel prior semantic image is constructed in the following manner: the component annotation information (including component position and component type) of the standard image is converted to obtain an information form capable of expressing the prior probability of the component on the image; a multi-channel gray image with the number of channels consistent with the total number of all component types in the complex equipment is initialized; the information form is superimposed on the multi-channel gray image to generate a multi-channel prior semantic image as the tensorized prior structure knowledge, and the prior semantic image of each channel represents the prior position and size of a certain component type in the standard image in the form of pixel value representing probability;

[0047] S2, using a pre-constructed multi-channel semantic image input branch, fusing the last two layers of high semantic feature images generated by the trained inspection image feature extraction network for each inspection image and the multi-channel prior semantic image corresponding to the inspection image to realize the enhancement of the prior structure knowledge to the inspection image features; wherein the multi-channel semantic image input branch is constructed by combining the Bottleneck module of Resnet and the deformable convolution operator;

[0048] S3, based on the fused feature image and the feature images output by other output layers of the inspection image feature extraction network, the component target recognition result is obtained through regression and classification.

[0049] It should be noted that the inspection images of each area on the complex equipment in the workshop are collected by taking pictures of the complex equipment in the workshop. As a preferred embodiment, the inspection images of each area on the complex equipment in the workshop are collected by the inspection robot, and the complex equipment in the workshop is inspected and photographed based on the 2D or 3D camera at the end of the inspection robot. For example, if the complex equipment is the bottom bogie of the motor train set parked in the maintenance workshop, the bottom bogie of the motor train set is inspected and photographed, the inspection robot is equipped with two mechanical arms, a left mechanical arm and a right mechanical arm, and the ends of the two mechanical arms are configured with precision scanning 3D cameras to realize high-resolution image collection of the bottom bogie of the motor train set.

[0050] In addition, since the manufacturing and installation of parts of the same complex equipment (such as a motor train set) comply with strict specifications, the spatial structure of the parts in the images taken at the same collection point of the same vehicle model is consistent, and the inspection images and the standard images are both taken and collected by the inspection robot at several specific positions. Therefore, a unique standard image can be mapped based on the collection position of the inspection image according to the position constraint. The prior semantic image representing the spatial position of the part corresponding to each inspection image is obtained from the standard image, which is used for subsequent part target recognition.

[0051] Regarding the implementation of mapping, the following methods can be used:

[0052] The mapping relationship between the inspection images and the standard images is reflected in the naming of the image files. The naming of the inspection images includes time information and position information of the shooting, while the naming of the standard images can only have position information. For example, the naming is as follows:

[0053] Standard image: 1#-abc, 2#-abb, etc.

[0054] Inspection image: 20230101-1#-abc, 20230101-2#-abb, etc.

[0055] The standard image corresponding to the inspection image 20230101-1#-abc is 1#-abc, and the standard image corresponding to the inspection image 20230101-2#-abb is 2#-abb. Therefore, any inspection image can obtain a unique standard image through position information.

[0056] Now, regarding the collection and storage of inspection images, the following implementation examples are given with the motor train set as an example:

[0057] The train vehicle bottom inspection robot detects the wheel set and axle through a positioning sensor, so that the inspection robot stops at a preset position. The 3D camera carried at the end of the robot collects inspection images of the motor train set at different positions on the maintenance track, and is named and stored according to time and position.

[0058] The acquisition process can be roughly divided into two main stages: in the first stage, the mechanical arm is retracted, the vehicle body is continuously moved from the first carriage to the last carriage, and the bogie and the bottom plate are acquired by the fast scanning camera carried by the vehicle body; in the second stage, the vehicle body moves from the last carriage to the first carriage, and the wheel set and the axle are detected by the positioning sensor during the movement, so that the vehicle body is stopped at the preset position, then the mechanical arm is raised and adjusted to the preset posture, and the high-precision camera carried by the end of the mechanical arm acquires the specific position precision scanning image.

[0059] It should be noted that the high-precision 3D camera carried by the end of the mechanical arm of the inspection robot can output RGB images and 3D point cloud information at the same time. It projects the image coded by the light machine to the measured object according to a certain rule and mode, and the coded pattern is modulated by the shape of the object surface to produce deformation. The structured light with deformation is captured by the area array camera at a fixed position, and the three-dimensional reconstruction of the object is performed through the positional relationship between the area array camera and the light source and the degree of structured light deformation. While generating 3D point cloud, depth information image matching with RGB image can also be generated.

[0060] Now regarding the creation of standard images, the following further explanation is given:

[0061] A set of standard images for EMU component target recognition is created. Since the position of the image acquired by the inspection robot is basically fixed, a complete set of standard images can be acquired for all acquisition points, which will be used to construct prior structural knowledge in component target recognition.

[0062] The standard image needs to be labeled using LabelImg, but the labeling specification is different from the inspection image used for training, that is, the standard image can not label all component targets appearing on the image, but only label the key targets on the image. The so-called key target refers to the target concerned by the subsequent task (such as defect detection task) based on component recognition. These targets have a clear view in the acquisition point, and are not distributed in the edge area of the image, so as to avoid the positioning error of the robot and the mechanical arm from making them disappear in the view.

[0063] It should be noted that the key target and the non-key target are for a single component, so a component type may have both key targets and non-key targets in the same standard image.

[0064] In addition, the multi-channel prior semantic image represents the prior semantics (including type, position and length-width information) of the target component in the standard image. In order to enable the multi-channel prior semantic image to correctly modulate the features in the high semantic feature image, the multi-channel prior semantic image needs to be combined with the low-resolution, high semantic feature image output by the feature extraction network to provide the accuracy of target recognition. Therefore, the application proposes a component target recognition method fusing prior structure knowledge.

[0065] As a preferred embodiment, the above information conversion mode is Gaussian probability ellipse semantic information extraction. The Gaussian probability ellipse semantics indicates the length, width, center point and other information of the target, which can ensure that the complete prior semantic distribution is extracted and the distribution curve is smooth.

[0066] As a preferred embodiment, the components on the complex equipment in the workshop are divided into fixed components and rotating moving components according to whether their positions are fixed, and different prior semantic image construction methods are used to construct the prior semantic images of the corresponding category channels. That is, for a standard image, the semantic images of all fixed components in the standard annotation and the semantic images of all rotating moving components in the standard annotation are constructed respectively; then the constructed fixed component semantic images and rotating moving component semantic images are merged to form the tensorized prior structure knowledge. The two types of components use inconsistent semantic image construction methods, which can improve the accuracy of complex component semantic image construction and better represent the semantics (including position information and length-width information) of the target in the standard image.

[0067] In constructing the multi-channel prior semantic image of each standard image, a multi-channel grayscale image needs to be initialized first, the pixel value of the multi-channel grayscale image is a preset value, and the value range of the pixel value is [0, 1], and the length and width of the multi-channel grayscale image is one-eighth of the standard image, so as to reduce the consumption of video memory and memory. The preset value of the pixel value in the initialization process can be a small value such as 0.01, 0.001, 0.002, and in this example, the preset value is selected as 0.01.

[0068] The prior semantic image of the corresponding category channel of each fixed component in a standard image is constructed by the following method:

[0069] S1, traverse each fixed component in the standard image, and generate a Gaussian ellipse two-dimensional semantic normal distribution of the fixed component according to its bounding box;

[0070] Specifically, the following method can be used:

[0071] The center of the bounding box is taken as the mean value, and the half width and half length of the bounding box are taken as the 2 times standard deviation in X and Y directions to generate a two-dimensional normal distribution with a peak value of 1.0 and X and Y independent, i.e. Gaussian ellipse, and the process is shown in the following formula:

[0072]

[0073]

[0074] x∈[μ x -3σ x ,μ x +3σ x ],y∈[μ y -3σ y ,μ y +3σ y ];

[0075] wherein, x min , x max , y min , y max represent the range of the target bounding box, μ and σ represent the mean and standard deviation of the normal distribution. For a pixel with position (x, y) on the semantic image, the value is assigned as SemanticValue(x, y). The value range of each target is 3σ range centered on μ;

[0076] S2, add the Gaussian ellipse two-dimensional semantic normal distribution of each fixed component in the corresponding position of the gray image of the corresponding category channel. If the Gaussian ellipse two-dimensional semantic normal distributions of the components of the same type overlap, the relative high value is taken for each pixel in the overlapping area.

[0077] As a preferred embodiment, the priori semantic image of the corresponding category channel of the moving component in each standard image is constructed in the following manner:

[0078] According to the motion trajectory of each type of component on the complex instrument, the type of the curve projected on the standard image by the trajectory and the formula are determined. Each moving component of the type in the standard image is traversed, the center of the bounding box of each moving component of the type is taken as the fitting point, the specific formula parameters are fitted according to the type of the curve and the formula, and the specific required curve formula is obtained.

[0079] The coordinates of the pole point of the required curve formula are calculated, and the ranges of x and y of the curve in the image are obtained. If the tangent line passing through a point on the curve is parallel to the x-axis or the y-axis, the point is called the pole point of the curve in the x-axis or y-axis direction.

[0080] According to the ranges of x and y of each curve in the image, the point set on the curve is calculated, and the points in the point set that are occluded by the moving trajectory of the moving component of the corresponding type in the standard image are deleted.

[0081] A one-dimensional normal distribution, i.e. a Gaussian normal circle, is produced in the vicinity of each curve with its curve point set as the center, as the semantic distribution of the corresponding type of moving part;

[0082] The semantic distribution of each type of rotating moving part is added to the corresponding position of the gray-scale image of the corresponding category channel.

[0083] For example, when the complex equipment in the workshop is a rail transit vehicle bogie, the prior semantic image of the corresponding channel of the rotating moving part in each standard image is constructed in the following manner:

[0084] S1. Each rotating moving part in the standard image is traversed, and the boundary box center of each type of part is taken as the fitting point to fit the conic curve of the motion trajectory of the part of this type, which is expressed as: F(x, y) = ax 2 +by 2 +cxy+dx+ey=1, and the type of the conic curve is determined by the discriminant Delta, Delta = b 2 -4ac. If Delta < 0, it indicates that the motion trajectory projection is an ellipse, and if Delta > 0, it indicates that the motion trajectory projection is a hyperbola. It should be noted that due to the characteristics of computer floating-point number calculation, Delta is almost impossible to be 0, and therefore Delta = 0 is not considered.

[0085] S2. According to the type of each conic curve, the coordinates of the pole point are obtained, and the ranges of x and y of the conic curve in the image are obtained. If the tangent line passing through a point on the conic curve is parallel to the x-axis or y-axis, the point is called the pole point of the conic curve in the x-axis or y-axis direction.

[0086] Specifically, the following method can be used:

[0087] The general equation of the conic curve is converted into the following form:

[0088] ax 2 +(by+d)x+(cy 2 +ey-1)=0;

[0089] cy 2 +(bx+e)y+(ax 2 +dx-1)=0;

[0090] Then the one-dimensional quadratic equations about x and y are obtained, and according to the discriminant of the one-dimensional quadratic equation, the condition for having and only having a unique real root is that the formula is:

[0091] (by+d) 2 -4a(cy 2 +ey-1)=0;

[0092] (bx+e) 2 -4c(ax 2 +dx-1)=0;

[0093] That is, a quadratic equation about y and x without unknowns is obtained, and the roots of the above formula are respectively named y r1 , y r2 and x r1 , x r2 . If the conic curve is an ellipse, the above condition formula has real roots, and the conic curve has extreme points in the y-axis and x-axis directions; if the conic curve is a hyperbola, the above condition formula has only one real root, and the other will get a pair of conjugate complex roots, that is, the hyperbola has an extreme point in the y-axis or x-axis direction;

[0094] S3, according to the range of x and y of each conic curve in the image, the point set on the conic curve is calculated, and the part of the point set corresponding to the type of the rotating moving part motion trajectory of the conic image is removed in the image;

[0095] Specifically, the point set is calculated, which can be calculated in the following way:

[0096] After limiting the real part of all roots within the coordinate range of the semantic image, the span y count and x count between the two roots in the corresponding coordinate axis direction are obtained:

[0097] y count =|y r1 -y r2 |;

[0098] x count =|x r1 -x r2 |;

[0099] If it is an ellipse, y count and x count are both greater than 0; if it is a hyperbola, one of them will be equal to 0 because of the conjugate complex roots. The larger one of y count and x count is selected to sample the corresponding coordinate axis to obtain the point set of the conic curve. For example, if y count >x count , 100 y values are generated linearly between y r1 and y r2 , and then the corresponding x values are obtained by substituting the conic curve equation, that is, the point set on the conic curve is obtained;

[0100] S4. In the vicinity of each conic curve, a one-dimensional normal distribution, i.e., a Gaussian positive circle, is generated with the conic curve point set as the center, as the semantic distribution of the corresponding type of rotating moving component;

[0101] The reason for using a Gaussian positive circle instead of an ellipse here is that the semantic distribution of the rotating moving component is generated by multiple components, and the radius of the distribution is the mean value of their scales; the average size of all N rotating moving component bounding boxes in the image is calculated, and one quarter of the average size is taken as the standard deviation of the one-dimensional normal distribution, as shown in the formula:

[0102]

[0103] All points in the conic curve point set are taken as the mean value of the one-dimensional normal distribution, σ is taken as the standard deviation, and 0.5 is taken as the peak value of the normal distribution. The one-dimensional normal distribution values within the 2σ range around all points in the point set are calculated. The Euclidean distance is taken as the independent variable for calculating the one-dimensional normal distribution. For a point P in the conic curve and a point Q within the 2σ range of P, the calculation formula of the semantic distribution value is:

[0104]

[0105] If the semantic distribution value is repeatedly generated at a point Q, the maximum value is taken. Here, the peak value of the normal distribution is set to 0.5 instead of 1.0, because compared with fixed components, rotating moving components can appear at any position in the trajectory, but the probability of each position appearing is smaller than the probability of the fixed component appearing. In other words, the "existence" of the rotating moving component on the image is "flattened" by the moving trajectory;

[0106] S5. The semantic distribution of each type of rotating moving component is added to the corresponding position of the gray-scale image of the corresponding class channel.

[0107] According to the prior semantic image construction method of the fixed component and the rotating moving component described above, the Gaussian ellipse semantic distributions generated by three different fixed components in the same standard image shown in Figure 2 Group 1 and the semantic distributions of the rotating moving bolts shown in Figure 2 Group 2, which were taken at two different angles, are combined to form the tensorized prior structural knowledge.

[0108] As a preferred implementation, the implementation of the above-mentioned fusion of the multi-channel semantic image input branch is as follows:

[0109] S1. Reduce the resolution of the multi-channel semantic image (the reduction factor is related to the amount of calculation), and then perform a 1x1 convolution to increase the dimension (only increase the number of channels, the purpose is to make the number of convolution tensor channels involved in subsequent calculation as much as possibleN upsample (for example, upsampling can be performed by using a deconvolutional layer with a step of 2) and then downsample (for example, downsampling can be performed by using a pooling layer with a step of 2) so as to make the multi-channel semantic image consistent with the resolution of the high semantic feature image output by the penultimate level of the inspection image feature extraction network;

[0110] S2, connecting the multi-channel semantic image obtained in step S1 and the high semantic feature image output by the penultimate level of the inspection image feature extraction network in the channel to obtain a tensor;

[0111] S3, making the tensor pass through a deformable convolutional operator (for example, a deformable convolutional operator containing a 3x3 convolutional kernel and two groups of offsets) to realize preliminary alignment and fusion of the multi-channel semantic image and the feature image, and then passing through K Bottleneck modules for feature extraction to obtain a first image feature fused with prior semantic;

[0112] S4, fusing the first image feature fused with prior semantic after maximum pooling with the high semantic feature image output by the antepenultimate level of the inspection image feature extraction network, and then passing through T Bottleneck modules for feature extraction and then passing through a 1x1 convolution to obtain a second image feature;

[0113] S5, replacing the first image feature and the second image feature with the output of the penultimate two levels of the original inspection image feature extraction network to input the regression and separation network to realize input and fusion of the prior semantic image in feature extraction, wherein M, N, K and T are positive integers.

[0114] For example, if the inspection image feature extraction network adopts the feature extraction network of Faster R-CNN FPN, a multi-channel semantic image input branch based on Faster R-CNN FPN is designed to fuse the prior structural knowledge constructed in the foregoing into target detection.

[0115] The multi-channel semantic image represents the prior semantic (including type, position and length-width information) of the target component in the standard image. In order to enable the multi-channel semantic image to correctly modulate the features in the high semantic feature image, the multi-channel semantic image needs to be combined with the low-resolution and high semantic feature image output by the feature extraction network. Therefore, the multi-channel semantic image input branch constructed in the embodiment takes the last two levels of output of the feature extraction network as the input tensor.

[0116] In order to more clearly illustrate the multi-channel semantic image input branch, the following example is given:

[0117] Firstly, the resolution of the inspection image feature extraction network is input into the branch with a semantic image input size of 1 / 8, and then a 1x1 convolution is used to upgrade the dimension, and then a pooling with a step of 2 is used to change the resolution to 1 / 16, and is connected with Conv4 in the channel to obtain a 1280-dimensional tensor; at this time, the 1280-dimensional tensor is processed through path ① (as shown in Figure 3 The path ① is a deformable convolution operator DCN (Deformable Convolution), which can be preferred, and the deformable convolution operator contains a 3x3 convolution kernel and two groups of offsets and the output dimension is 1024, and the principle of the deformable convolution operator can be simply represented by the formula:

[0118]

[0119] That is, an offset Δp k is calculated for each pixel of the input image, so that the shape of the convolution kernel acting on the input image breaks the regular square receptive field. The offset Δp k is actually predicted by a normal convolution inside the DCN.

[0120] The significance of using DCN here can be explained as follows: there may be a relative change between the shooting positions of the test image and the standard image, and it is expected to correct this relative position relationship through the offsets of the DCN to realize approximate position registration of the features.

[0121] As a comparison, the 1280-dimensional tensor is processed through path ② as shown in Figure 3 , that is, only a 1x1 normal convolution is used to transform the channel number, and then the effectiveness of using DCN in the multi-channel semantic image input branch can be observed through the results of the comparison experiment, and the results of the comparison experiment are shown in Table 2.

[0122] After passing through path ① or ②, the features are extracted through 4 Bottleneck modules to obtain features Conv4_S fused with prior semantic, and then the last layer of high semantic feature image (Conv5 feature layer) is fused to obtain Conv5_S. Conv4_S and Conv5_S are input into FPN instead of the original Conv4 and Conv5 to realize the input and fusion of prior structural knowledge in feature extraction.

[0123] For example, if the inspection image feature extraction network adopts the feature extraction network of YOLO v7, a multi-channel semantic image input branch based on YOLO v7 is designed to fuse the prior structural knowledge constructed in the foregoing into target detection.

[0124] For the multi-channel semantic image input branch based on YOLO v7 as shown in Figure 4 , the multi-channel semantic image input branch andFigure 3 The overall structure of the Faster R-CNN FPN-based multi-channel semantic image input branch shown has little difference, so similarly, path ① with DCN is better than path ②, the only difference is that due to the difference in the number of output channels of the inspection image feature extraction network, the relevant parameters of the input module are modified accordingly.

[0125] Next, Faster R-CNN FPN and YOLO v7 two target recognition networks are studied, and two path-selected multi-channel semantic image input branches are provided, and through comparative experiments, it is shown that path ① is better than path ②. Therefore, six experimental groups are set up to explore the influence of the target recognition neural network input module on the performance of the target detection framework and to prove the role of DCN in the branch.

[0126] The experimental groups are as follows:

[0127] The first group, Faster R-CNN FPN (abbreviated as FRCNNF), is a primitive target recognition network;

[0128] The second group, FRCNNF-SI, uses a target recognition neural network input module without DCN, corresponding to path ②; Figure 3

[0129] The third group, FRCNNF-DCN SI, uses a target recognition neural network input module with DCN, corresponding to path ①; Figure 3

[0130] The fourth group, YOLO v7, is a primitive target recognition network;

[0131] The fifth group, YOLO v7-SI, uses a target recognition neural network input module without DCN, corresponding to path ②; Figure 4

[0132] The sixth group, YOLO v7-DCN SI, uses a target recognition neural network input module with DCN, corresponding to path ①. Figure 4

[0133] Among them, for experimental groups based on the same target recognition network, the network parameters and training parameter settings are completely consistent.

[0134] ​​​​For the three experimental groups (the first group to the third group) based on Faster R-CNN FPN, the input scale of 1536x960 is used, the pre-training Backbone weight on ImageNet using Resnet50 is used, the weight of the Backbone is frozen in the initial stage, other network parameters are trained for 40 epochs, the Adam optimizer is used, the initial learning rate is set to 0.0001, and the decay rate is set to 0.96; then the Backbone weight is unfrozen, and 60 epochs of training are continued, the learning rate is reset to 0.0001 and the decay begins.

[0135] For the three experimental groups (the fourth group to the sixth group) based on YOLO v7, the input scale of 640x640 is used, the Backbone part of the official pre-training weight is used, and the Backbone is frozen in the initial stage for 40 epochs of training, the SGDM optimizer is used, the initial learning rate is set to 0.001, the momentum parameter momentum is set to 0.937, and the weight decay is set to; then the Backbone weight is unfrozen, and 60 epochs of training are continued, the learning rate is reset to 0.001 and the decay begins.

[0136] A certain number of labeled inspection images and standard images are used as a data set, and the training set and the test set are divided according to a certain proportion, and the training set is used to train the target recognition network.

[0137] After training, the performance of the target recognition network is evaluated by using relevant evaluation criteria.

[0138] The target prediction results used to calculate the evaluation criteria are divided into four categories, which are:

[0139] TP: True Positive, indicating the number of successfully hit prediction boxes of the true value, where the success of the hit is usually determined according to the consistency of the class label, and whether the IoU of the prediction box and the true value box is greater than a certain threshold.

[0140] FP: False Positive, indicating the number of prediction boxes that do not successfully hit the true value.

[0141] TN: True Negative, indicating the successfully ignored background class, which will not be used in target detection.

[0142] FN: False Negative, indicating the number of true values that are not hit by the prediction box.

[0143] The evaluation criteria include the following indicators:

[0144] Precision represents the proportion of correct predictions among all predictions, as shown in the following formula:

[0145]

[0146] Recall represents the proportion of true values that are successfully predicted, as shown in the following formula:

[0147]

[0148] In addition, the single-class average precision AP represents the area under the Precision-Recall (P-R) curve of a certain class. First, sort the prediction boxes of the class according to the confidence, set an initially empty prediction box set, and gradually add the prediction box with the highest confidence to the set, then calculate the Precision and Recall of the set at this time, and a series of points on the P-R curve can be obtained. Finally, select several Recall sampling coordinates on the P-R curve, obtain the corresponding Precision by interpolation method, fit the curve, and accumulate the area to obtain AP.

[0149] The average precision mAP of all classes represents the arithmetic mean of AP of all classes.

[0150] Table 1 below shows the performance of the six experimental groups according to the evaluation criteria, and the training set loss and test set loss of each experimental group model have converged, and the final model is selected as the test loss minimum. Among the evaluation indicators, and respectively represent the mAP values of the training set when the IoU threshold is set to 0.5 and 0.75 when calculating TP, the higher the IoU threshold, the higher the requirement for the boundary box regression accuracy. and respectively represent the mAP values of the test set. Params and FPS evaluate the size and calculation speed of the algorithm model.

[0151] Table 1: Overall performance of the algorithm of each experimental group

[0152]

[0153] From the experimental results, for the three experimental groups based on FRCNNF, using the DCN SI branch of path ①, the mAP of the training set and test set is improved by 9.41% and 7.33% respectively relative to the baseline when the IoU threshold is 0.5, and by 8.33% and 5.13% respectively relative to the SI branch of path ②; when the IoU threshold is 0.75, the mAP of the training set and test set is improved by 5.45% and 16.6% respectively relative to the baseline, and by 4.79% and 16.14% respectively relative to the SI branch of path ②.

[0154] For the three experimental groups based on YOLO v7, the mAP of the training set and the test set relative to the baseline increased by 3.32% and 9.48% respectively at an IoU threshold of 0.5 using the DCN SI branch of path ①, and increased by 2.26% and 4.12% respectively relative to the SI branch of path ②; at an IoU threshold of 0.75, the mAP of the training set and the test set relative to the baseline increased by 8.89% and 14.5% respectively, and increased by 7.3% and 9.61% respectively relative to the SI branch of path ②.

[0155] It can be seen that for both the two-stage FRCNN F and the one-stage YOLO v7 target recognition networks, the use of the multi-channel semantic image input branch improves the overall accuracy and the accuracy of the bounding box regression. At the same time, from the comparison between the SI branch and the DCN SI branch, it can be seen that the use of the deformable convolution operator plays an important role in the semantic image input branch, greatly improving the improvement effect of the semantic image input branch.

[0156] Although increasing the multi-channel semantic image input branch will increase the parameter quantity of the algorithm model and slightly reduce the calculation speed, the component recognition task is performed on a server with sufficient computing resources and does not require real-time performance, and the component recognition task is related to the accuracy of subsequent tasks (such as maintenance tasks), so it is valuable to sacrifice some model simplification and speed to obtain higher component recognition accuracy in this task.

[0157] The following Table 2 also shows the recognition accuracy of the six experimental groups for several types of EMU bogie key components in the test set. Among them, AP1 represents the accuracy of the bolt, AP2 represents the accuracy of the check wire, and AP3 represents the accuracy of the brake pad.

[0158] Table 2 Recognition accuracy of key components by algorithm of each experimental group

[0159]

[0160]

[0161] Therefore, the DCN SI branch greatly improves the recognition accuracy of key components. The key component recognition accuracy of FRCNN F-DCN SI is improved by 4% to 8% compared with the baseline when the IoU threshold is 0.5, and is improved by 4% to 15% compared with the baseline when the IoU threshold is 0.75. The key component recognition accuracy of YOLO v7-DCN SI is improved by 4% to 20% compared with the baseline when the IoU threshold is 0.5, and is improved by 5% to 11% compared with the baseline when the IoU threshold is 0.75. The introduction of the multi-channel semantic image input branch enhances the identification performance of the target detection framework on the key components of the bogie of the motor train unit, and lays a solid foundation for subsequent component identification-based tasks.

[0162] For industrial equipment similar to the bogie of the rail transit vehicle that can form a standard image, the method has certain applicability and similar applicability to the bogie of the motor train unit.

[0163] Embodiment two

[0164] A method for constructing a component target recognition model, comprising:

[0165] The component target recognition network is built, including an inspection image feature extraction network, a multi-channel semantic image input branch, and a regression and separation network. A training sample set is constructed, the component target recognition network is trained, and a component target recognition model integrating prior structural knowledge is obtained, which is used to execute the component target recognition method integrating prior structural knowledge as described in Embodiment One.

[0166] The inspection image feature extraction network is used to extract a feature image of an inspection image, and the last two layers output a high semantic feature image. The multi-channel semantic image input branch is used to fuse the last two layers of high semantic feature images generated by the inspection image feature extraction network for each inspection image and the corresponding multi-channel prior semantic image of the inspection image, so as to enhance the prior structural knowledge on the inspection image features. The multi-channel semantic image input branch is constructed by combining the Bottleneck module of Resnet and the deformable convolution operator. The regression and classification network is used to obtain a component target recognition result based on the fused feature image and the feature images output by other output layers of the inspection image feature extraction network.

[0167] The collection of inspection images and standard images, the construction of prior structural knowledge, and the construction of the multi-channel semantic image input branch are not described again here.

[0168] Embodiment three

[0169] The application further provides a computer readable storage medium, which comprises a stored computer program, wherein the computer program controls a device where the storage medium is located to perform the component target recognition method of fusing prior structure knowledge and / or the component target recognition model construction method when the computer program is run by a processor.

[0170] The related technical solutions are the same as those in Embodiment One and Embodiment Two, and will not be described herein again.

[0171] Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A component target recognition method incorporating prior structural knowledge, characterized by, The application comprises the following technical solutions: Collecting inspection images of each area on complex equipment in a workshop, determining a standard image to be collected based on the collection position of each inspection image, and using position constraint mapping to obtain a multi-channel prior semantic image of the corresponding area obtained from the standard image, wherein the multi-channel prior semantic image is constructed in the following manner: information conversion is performed on the component annotation information of the standard image to obtain an information form capable of expressing the prior probability of the component on the image; a multi-channel gray image with a channel number consistent with the total number of all component types in the complex equipment is initialized; and the information form is superimposed on the multi-channel gray image to generate a multi-channel prior semantic image as the tensorized prior structural knowledge, and the prior semantic image of each channel represents the prior position and size of a certain component type in the standard image in the form of a probability represented by a pixel value. The last two layers of high semantic feature images generated by the trained inspection image feature extraction network and the multi-channel prior semantic image corresponding to each inspection image are fused by using a pre-constructed multi-channel semantic image input branch to realize the enhancement of the prior structural knowledge on the inspection image features, wherein the multi-channel semantic image input branch is constructed by combining a Bottleneck module of Resnet and a deformable convolution operator. Based on the fused feature image and the feature images output by other output layers of the inspection image feature extraction network, the component target recognition result is obtained through regression and classification. The components on the complex equipment in the workshop are divided into fixed components and mobile components according to whether their positions are fixed, and different prior semantic image construction methods are used to construct the prior semantic images of the corresponding category channels. The prior semantic image of the corresponding category channel of each fixed component in a standard image is constructed in the following manner: Each fixed component is traversed, and a Gaussian ellipse two-dimensional semantic normal distribution of the fixed component is generated according to the bounding box thereof. The Gaussian ellipse two-dimensional semantic normal distribution of each fixed component is added to the corresponding position of the gray image of the corresponding category channel, and if the Gaussian ellipse two-dimensional semantic normal distributions of the same type of components overlap, the relative high value is taken for each pixel in the overlapping area.

2. The component target recognition method according to claim 1, characterized in that, The information conversion method is Gaussian probability ellipse semantic information extraction.

3. The component target recognition method according to claim 1, characterized by, The inspection images of each area on the complex equipment in the workshop are collected by an inspection robot, and the complex equipment in the workshop is inspected and photographed by a 2D or 3D camera at the end of the inspection robot.

4. The component target recognition method according to claim 1, characterized by, The pixel value of the multi-channel gray image is a preset value, and the value range of the pixel value is [0, 1], and the length and width of the multi-channel gray image are down-sampled to one-eighth of the standard image.

5. The component target recognition method according to claim 1, characterized in that, The prior semantic image of the corresponding category channel of each mobile component in a standard image is constructed in the following manner: According to the motion trajectory of each type of component on the complex instrument, the type of curve and the formula of the trajectory projected on the standard image are determined; each moving component of the type in the standard image is traversed, the center of the bounding box of each moving component of the type is taken as a fitting point, and the specific formula parameters are fitted according to the type of curve and the formula to obtain a specific required curve formula; The coordinates of the pole point of the required curve formula are calculated to obtain the x and y ranges of the curve in the image, wherein if the tangent line passing through a point on the curve is parallel to the x-axis or the y-axis, the point is called the pole point of the curve in the x-axis or y-axis direction; According to the x and y ranges of each curve in the image, a point set on the curve is calculated, and the points in the point set that are shielded in the image by the moving trajectory of the moving component of the corresponding type in the standard image are deleted; A one-dimensional normal distribution, i.e., a Gaussian normal circle, is generated in the vicinity of each curve with the curve point set as the center, serving as the semantic distribution of the moving component of the corresponding type; The semantic distribution of each type of moving component is added to the corresponding position of the gray image of the corresponding category channel.

6. The component target recognition method according to any one of claims 1 to 5, characterized by, The implementation manner of the multi-channel semantic image input branch for the fusion is: S1, reducing the resolution of the multi-channel semantic image, then upgrading the dimension through a 1*1 convolution, and then down-sampling, so that the resolution of the multi-channel semantic image is consistent with that of the high semantic feature image output by the second last level of the inspection image feature extraction network; S2, connecting the multi-channel semantic image obtained in step S1 with the high semantic feature image output by the second last level of the inspection image feature extraction network in the channel to obtain a tensor; S3, making the tensor pass through a deformable convolution operator to realize the preliminary alignment and fusion of the multi-channel semantic image and the feature image, and then performing feature extraction through K Bottleneck modules to obtain a first image feature fused with prior semantics; S4, fusing the first image feature fused with prior semantics with the high semantic feature image output by the last level of the inspection image feature extraction network after maximum pooling; then performing feature extraction through T Bottleneck modules, and then performing a 1*1 convolution to obtain a second image feature; S5, replacing the output of the last two levels of the original inspection image feature extraction network with the first image feature and the second image feature to input the regression and separation network to realize the input and fusion of the prior semantic image in feature extraction, wherein K and T are positive integers.

7. A method of constructing a component target identification model, characterized by, It comprises: building a component target recognition network, including an inspection image feature extraction network, a multi-channel semantic image input branch, and a regression and classification network; constructing a training sample set, training the component target recognition network, and obtaining a component target recognition model fused with prior structural knowledge, which is used to perform a component target recognition method fused with prior structural knowledge according to any one of claims 1 to 6; wherein the inspection image feature extraction network is used to extract the feature image of the inspection image, and the last two layers output a high semantic feature image. The multi-channel semantic image input branch is used for fusing the last two layers of high semantic feature images generated by the inspection image feature extraction network for each inspection image and the multi-channel prior semantic image corresponding to the inspection image, so as to realize the enhancement of the prior structure knowledge on the inspection image features; wherein the multi-channel semantic image input branch is constructed by combining a bottleneck module of Resnet and a deformable convolution operator; The regression and classification network is used for obtaining the component target recognition result based on the fused feature image and the feature images output by other output layers of the inspection image feature extraction network.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer program controls the device where the storage medium is located to execute the component target recognition method of fusing prior structure knowledge and / or the construction method of the component target recognition model according to any one of claims 1 to 6 and 7 when the computer program is run by the processor.

Images