A mobile inspection method and device, computer equipment and storage medium

Abstract

The application discloses a kind of mobile inspection method, device, computer equipment and storage medium, the method includes: by fixed camera fixed image is obtained, and by mobile camera obtains the mobile image of different angles at the same point position;Different angle mobile image is spliced, and the spliced image is obtained, and the fixed image is matched with scene, and corresponding scene information is obtained;Based on the scene information, target object reasoning is carried out, and target object reasoning information containing target object to be detected is obtained;The scene information and target object reasoning information are combined as prompt information, and based on the prompt information, the target object is modelled by classification model, and corresponding detection information is obtained.The application can solve the problem of incomplete detection caused by the visual angle blind area of traditional fixed camera, improve the calibration accuracy of detection point, avoid the time-consuming and error-prone of manual configuration algorithm scene, achieve efficient and intelligent automatic inspection effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer software, in particular to a mobile inspection method and device, computer equipment and storage medium. BACKGROUND

[0002] In the park safety detection scene, the traditional fixed camera is difficult to effectively identify potential safety hazards under the blind area due to the existence of the visual angle blind area, and cannot meet the comprehensive detection demand. To solve this problem, a mobile inspection robot emerges as the times require, which can cover the detection area from multiple angles by moving to multiple positions, and theoretically can realize the non-blind area detection task. However, this scheme is prone to problems such as point position deviation and disorder in actual application, and how to accurately calibrate the detection point position is a problem faced by this scheme.

[0003] At the same time, the algorithm scene configuration of the fixed camera still depends on manual operation, which not only consumes a lot of time cost, but also often causes algorithm matching errors due to human operation errors, further affecting the detection efficiency and accuracy. In summary, how to realize the accurate inspection and detection process automation of the mobile robot multiple point positions has become a core challenge faced by the technical personnel in the field. With the rapid development of artificial intelligence algorithms, how to integrate machine learning, deep learning algorithms and large models into the mobile inspection system to solve the above problems and realize the automatic inspection of multiple point positions and multiple angles in the park is a problem that the technical personnel in the field need to solve. SUMMARY

[0004] The embodiments of the present application provide a mobile inspection method and device, computer equipment and storage medium, aiming to realize efficient and intelligent automatic inspection effect.

[0005] In a first aspect, the embodiments of the present application provide a mobile inspection method, comprising:

[0006] acquiring a fixed image through a fixed camera, and acquiring a mobile image of the same point position and different angles through a mobile camera;

[0007] stitching the mobile images of different angles to obtain a stitched image, and performing scene matching on the stitched image and the fixed image to obtain corresponding scene information;

[0008] performing target object reasoning based on the scene information to obtain target object reasoning information containing a target object to be detected;

[0009] combining the scene information and the target object reasoning information into prompt information, and performing model reasoning on the target object through a classification model based on the prompt information to obtain corresponding detection information.

[0010] In a second aspect, an embodiment of the present application provides a mobile inspection device, comprising:

[0011] an image acquisition unit configured to acquire a fixed image through a fixed camera and acquire a mobile image of the same point at different angles through a mobile camera;

[0012] a splicing and matching unit configured to splice the mobile images at different angles to obtain a spliced image, and perform scene matching on the spliced image and the fixed image to obtain corresponding scene information;

[0013] a first inference unit configured to perform target inference based on the scene information to obtain target inference information containing a target to be detected;

[0014] a second inference unit configured to combine the scene information and the target inference information into prompt information, and perform model inference on the target through a classification model based on the prompt information to obtain corresponding detection information.

[0015] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the mobile inspection method according to the first aspect when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the mobile inspection method according to the first aspect.

[0017] The embodiment of the present application provides a mobile inspection method and device, computer equipment and a storage medium, the method comprises the following steps: acquiring a fixed image through a fixed camera, and acquiring a mobile image of the same point with different angles through a mobile camera; splicing the mobile images with different angles to obtain a spliced image, and performing scene matching on the spliced image and the fixed image to obtain corresponding scene information; performing target object reasoning based on the scene information to obtain target object reasoning information containing a target object to be detected; combining the scene information and the target object reasoning information into prompt information, and performing model reasoning on the target object through a classification model based on the prompt information to obtain corresponding detection information. The embodiment of the present application effectively solves the incomplete detection problem caused by the visual angle blind area of the traditional fixed camera by fusing the multi-source image data of the fixed camera and the mobile camera, and simultaneously greatly improves the calibration accuracy of the detection point by using the image splicing and scene matching technology, thereby avoiding the time-consuming and error-prone nature of manual algorithm scene configuration. Moreover, the embodiment of the present application realizes intelligent reasoning and detection of the specified target object through the deep application of machine learning and deep learning algorithms, thereby significantly improving the inspection efficiency and accuracy, and achieving the effect of efficient and intelligent automatic inspection. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 A flowchart of a mobile inspection method according to an embodiment of the present application is provided.

[0020] Figure 2 A sub-flowchart of a mobile inspection method according to an embodiment of the present application is provided.

[0021] Figure 3 Another sub-flowchart of a mobile inspection method according to an embodiment of the present application is provided.

[0022] Figure 4 A schematic block diagram of a mobile inspection device according to an embodiment of the present application is provided.

[0023] Figure 5 A sub-schematic block diagram of a mobile inspection device according to an embodiment of the present application is provided.

[0024] Figure 6 Another sub-schematic block diagram of a mobile inspection device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0026] It should be understood that the terms "comprising" and "including" as used in the specification and the appended claims indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0027] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0028] It should be further understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0029] Please see the following Figure 1 The method for mobile inspection provided by the embodiments of the present application specifically comprises steps S101-S104.

[0030] Step S101, acquiring a fixed image through a fixed camera and acquiring mobile images of the same point at different angles through a mobile camera;

[0031] Step S102, stitching the mobile images at different angles to obtain a stitched image, and performing scene matching on the stitched image and the fixed image to obtain corresponding scene information;

[0032] Step S103, performing target object reasoning based on the scene information to obtain target object reasoning information containing a target object to be detected;

[0033] Step S104, combining the scene information and the target object reasoning information into prompt information, and performing model reasoning on the target object through a classification model based on the prompt information to obtain corresponding detection information.

[0034] In this embodiment, first, the fixed image and the moving image are acquired by the fixed camera and the moving camera respectively. Then, the moving image is spliced to generate a spliced image, and scene matching is performed with the fixed image to extract scene information. Based on the scene information, further inference of the related information of the target object is performed. Finally, the scene information and the inference information of the target object are integrated into prompt information, the target object is inferred and detected by a classification model, and finally the detection result is obtained.

[0035] The embodiment effectively solves the incomplete detection problem caused by the visual angle blind area of the traditional fixed camera by fusing the multi-source image data of the fixed camera and the moving camera. At the same time, by using image splicing and scene matching technology, the calibration accuracy of the detection point is greatly improved, and the time-consuming and error-prone of manual configuration of algorithm scene are avoided. Moreover, by deep application of machine learning and deep learning algorithm, the embodiment realizes intelligent inference and detection of the specified target object, significantly improves the inspection efficiency and accuracy, and achieves efficient and intelligent automatic inspection effect.

[0036] In actual application, when acquiring the fixed image, video frame extraction operation can be performed on the video stream data of the accessed monitoring camera to acquire all camera pictures, and the detection area coordinate information and the belonging scene information are retained, and then the pictures can be arranged in the form of a list, for example, as shown in Table 1 below:

[0037] Table 1

[0038]

[0039] In an embodiment, as shown in Figure 2 The step S102 includes steps S201-S205.

[0040] In step S201, each moving image is standardized, and image features are extracted from the standardized moving image.

[0041] In this step, first, the size of the input multiple moving images (img0, img1, …, img n ) of the same point taken by the moving robot at different angles are uniformly adjusted to 640x480, and normalized to eliminate the influence of light difference on feature extraction. Then, ResNet-18 can be used as a backbone network for feature extraction. ResNet-18 has 18 layers of trainable parameters, including 16 convolutional layers and 2 fully connected layers, and is divided into 5 stages. Among them:

[0042] The input layer is a 7x7 convolution with a step of 2 and a maximum pooling layer for extracting low-level features.

[0043] Residual block group 1 has 4 layers in total, each layer is composed of 2 basic residual blocks and 3*3 convolution, used for stacking features;

[0044] Residual block group 2 has 4 layers in total, each layer is composed of 2 basic residual blocks and 3*3 convolution, used for lifting the number of channels;

[0045] Residual block group 3 has 4 layers in total, each layer is composed of 2 basic residual blocks and 3*3 convolution, used for deepening features;

[0046] Residual block group 4 has 4 layers in total, each layer is composed of 2 basic residual blocks and 3*3 convolution, used for advanced feature extraction;

[0047] The output layer has 2 layers, which are composed of global average pooling and fully connected layer.

[0048] In actual application, in order to extract features, the fully connected layer can be removed, and the last output feature of residual block group 4 is taken as the final feature extraction result, that is, the image feature.

[0049] Step S202, coordinate coding processing is performed on the image feature to obtain a first target feature with position information;

[0050] This step injects spatial position information into the extracted image feature, so that the model can perceive geometric relationships. Specifically, it includes:

[0051] (1) Grid coordinate generation, that is, generating relative coordinates (x, y) on the image feature and normalizing them to the range of [-1, 1];

[0052] (2) Sine-cosine coding, that is, using Transformer position encoding to map the coordinates to a high-dimensional space;

[0053] (3) Feature fusion, that is, adding the coordinate coding to the image feature to obtain the first target feature with position information.

[0054] Step S203, context association is performed on the target feature by alternately stacking self-attention and cross-attention to obtain a second target feature;

[0055] This step models the context association between features by alternately stacking self-attention and cross-attention, which specifically includes:

[0056] (1) Feature serialization, which flattens the two-dimensional feature map into a sequence according to the following formula:

[0057] ;

[0058] In the formula, F is the first target feature with position information, represents the flattened feature, and N*C represents the number of feature points;

[0059] (2) Self-attention processing, respectively, for each image feature sequence of the application multi-head self-attention, learning within the image features associated:

[0060] ;

[0061] In the formula, Q, K, V ∈ R N×dk Query, key, value matrix, dk = C / num_heads (the number of heads is usually 8), the output is processed through the residual connection and LayerNorm;

[0062] (3) Cross-attention, through cross-attention processing, to establish the correlation between the features of the images, according to the following formula using bidirectional calculation:

[0063] ;

[0064] In the formula, The feature points of the 0th picture are taken as Q queries, And The feature points of the first picture are taken as keys and values, respectively, F0 ' The feature points after cross-attention calculation are represented by F0

[0065] (4) Module stacking, i.e. repeating the self-attention and cross-attention processing structure for 4 layers, each layer is followed by a fully connected network to enhance the nonlinear expression ability.

[0066] Step S204, calculate the similarity of the second target feature, and perform mutual nearest neighbor matching according to the calculation result of the similarity to obtain an initial matching pair;

[0067] This step normalizes the features output by the Transformer, and then calculates the similarity using the cosine similarity formula, as follows:

[0068] ;

[0069] In the formula, The similarity between the i-th feature point of img0 and the j-th feature point of img1 is represented by f i ' And f j ' The i-th feature point of the 0th picture after cross-attention and the j-th feature point of the 1st picture after cross-attention are represented by F0

[0070] Then mutual nearest neighbor matching is performed, i.e. screening the initial matching pair:

[0071] For each i, find i.e. the nearest neighbor of i in img0;

[0072] For each j, find i.e. the nearest neighbor of j in img1;

[0073] If and and ( is a threshold, usually set to 0.2), then is the initial matching pair.

[0074] Step S205, geometric constraint filtering is performed on the initial matching pair to obtain a filtered reliable matching pair, and then each of the moving images is spliced into the spliced image based on the reliable matching pair.

[0075] This step uses the RANSAC algorithm to filter the matching that does not meet the geometric constraint, for example, for the matching point (p0, p1) (p0 is the point of the 0th image, and p1 is the point of the 1st image), the constraint of the fundamental matrix F is needed: so that where the fundamental matrix F is a geometric parameter of two images, which describes the internal relationship between the projection points of three-dimensional space points in two images, that is, to ensure that the matching points meet the geometric rules of camera imaging. The implementation process of the RANSAC algorithm is as follows:

[0076] (1) Randomly select 8 point pairs from the initial matching points (p0, p1), calculate the fundamental matrix F, so that ;

[0077] (2) Verify all matching point pairs by the fundamental matrix F obtained in the first step, and count the number of point pairs that meet P1ᵀFp0 ≈ 0 (the number of inliers);

[0078] (3) Repeat the sampling-calculation-verification process (that is, steps (1) and (2)) to keep all inliers corresponding to the F with the largest number of inliers as the filtered reliable matching points;

[0079] (4) Take out the pixel gray value in the local 3*3 window range around the filtered reliable matching points, and then optimize the coordinates of the pixel points through the bilinear interpolation pixel points, so that the matching points are closer to the true corresponding positions, and finally the coordinate position information of the fused pixel points is obtained, that is, the pixel points of the whole image are obtained, and the image is completed splicing;

[0080] (5) If the number of input images is greater than 2, repeat the above process.

[0081] In the embodiment, when the spliced image is acquired, the LoFTR algorithm is used to splice pictures of different angles at the same point position captured by the mobile robot to form a complete detection area, so that similarity calculation is performed with a fixed camera information list after splicing is completed, and scene information to which the pictures captured by the mobile robot belong is obtained.

[0082] In an embodiment, the step S102 further includes:

[0083] The ResNet-18 network is used to perform similarity calculation on the spliced image and the fixed image.

[0084] The fixed image with the highest similarity score is taken as a matching image.

[0085] Corresponding belonging scene and detection area coordinate information are extracted according to the matching image, and are summarized as the scene information.

[0086] In the embodiment, when scene matching is performed, the ResNet-18 network is used to perform similarity calculation on the spliced image (i.e., the mobile image after splicing) and the fixed camera picture, and the fixed camera image with the highest similarity score is taken as a matching image, and corresponding belonging scene and detection area coordinate information are extracted.

[0087] In an embodiment, the step S103 includes:

[0088] The scene information is encoded to obtain corresponding scene encoding.

[0089] The support vector machine model is used to classify and predict the scene encoding, and a result of the classification and prediction is set as the target object reasoning information.

[0090] The construction process of the support vector machine model includes:

[0091] A training data set is constructed, wherein the training data set includes a plurality of training target objects and corresponding training scene data.

[0092] Based on the training target objects, a one-to-one manner is used to set a classifier, and a radial basis function and a decision function are set to build an initial support vector machine framework.

[0093] The initial support vector machine framework is trained by using the training scene data to obtain the support vector machine model.

[0094] After obtaining the spatial scene, the corresponding target detection needs to be started. For example, in the community entrance scene, the target to be detected may be personnel, vehicles, garbage disposal, fire extinguishers and the like. Therefore, the embodiment realizes automatic target detection through target recommendation processing. The embodiment first encodes the scene, and then infers the target that may appear in the scene. Since the existence of each target is independently judged, it can be converted into n independent binary classification problems (corresponding to n targets) by an SVM model (support vector machine model). Each classification task is completed by an SVM model.

[0095] In actual application, the input is a 4-bit binary scene code, for example, the community entrance is 0001, which is converted into a feature vector x e R4, that is, [0, 0, 0, 1]. The output is the existence of each target, for example, 0 represents nonexistence, and 1 represents existence. Finally, it is combined into an n-bit binary code.

[0096] When constructing the SVM model, the embodiment adopts a one-to-one method to train an SVM classifier for each target. For example, classifier 1: judge whether “personnel” exists; classifier 2: judge whether “vehicle” exists; classifier 3: judge whether “fire extinguisher” exists; classifier 4: judge whether “garbage can” exists; …; and classifier n: judge whether “manhole cover” exists.

[0097] For the feature space and kernel function of the model, the input feature dimension is 4-dimensional code. According to the association relationship between the scene and the target, a radial basis function is selected. By mapping the low-dimensional feature to a high-dimensional space, the nonlinear correlation is captured. The kernel function is as follows:

[0098] ;

[0099] Among them, and respectively represent two input feature vectors, such as is the feature vector [0, 0, 0, 1] of “community entrance”, is the feature vector [0, 0, 1, 0] of “underground parking lot”, and γ is a kernel parameter. K represents a radial basis function, that is, the similarity between two feature vectors x i and x j in a high-dimensional space is calculated. Exp represents a natural exponential function, which is used to convert the square of the feature vector difference into similarity. The smaller the difference is, the closer the exp result is to 1, and the higher the similarity is.

[0100] For the decision function of the model, each SVM classifier obtains a decision hyperplane through learning. For the input feature x, the output is:

[0101] ;

[0102] wherein, is the weight of support vector, is the label of training sample, b is the bias term. If f(x) = 1, it is determined that the target object exists; if f(x) = -1, it is determined that the target object does not exist, f(x) represents the decision function of the SVM classifier, the input is the feature x to be judged, and the output result is converted by the sign function and used to determine whether the target object exists. The sign represents the sign function, which extracts the sign of the result inside the expression of f(x): if the result is positive, the sign outputs 1 (determining that the target object exists); if the result is negative, the sign outputs -1 (determining that the target object does not exist).

[0103] Then the model is trained. The construction of the training sample can be as shown in Table 2, and the scene information and target object information are labeled:

[0104] Table 2

[0105]

[0106] 1 in Table 2 represents existence, and 0 represents non-existence. After the labeling is completed, the data set is divided in a manner that 70% is used for training and 30% is used for verification.

[0107] Based on the constructed training data set, for each target object binary classification task, the SVM is trained with the sample corresponding to the label. The principle of the support vector machine is to divide the sample points in the original space or high-dimensional space to form a hyperplane. The formula for dividing the hyperplane is:

[0108] ;

[0109] In the formula, is the normal vector, b is the displacement term, X represents the input feature vector, and represents the original features of the sample (such as the 4-dimensional encoding of the scene).

[0110] In order to more accurately divide the correct class and find the appropriate so that the following formula is established:

[0111] ;

[0112] ;

[0113] In the formula, is the input data, represents the true label of the i-th training sample, and the value is only +1 or -1, and n represents the number of input data.

[0114] The data is mapped into high-dimensional space by a radial kernel function, whose formula is as follows:

[0115] ;

[0116] Considering that there are cases of long distance or unclear in the camera picture, so that the data often appear abnormal, in order to improve the effect of classification, reduce the influence of abnormal data on the prediction result, the slack variable is introduced:

[0117] ;

[0118] ;

[0119] In the formula, C is the penalty factor, is the slack variable. Here, the value of C and will directly affect the prediction effect, so before training, initialize C and parameter , and then select the mean absolute error (MAE) as the loss function of the training model, whose formula is as follows:

[0120] ;

[0121] In the formula, n represents the number of samples, represents the predicted value of the model, represents the true label of the i-th training sample.

[0122] The method of grid search can be used to select C and parameter one by one, and after each training, MAE and C, are counted respectively. When the MAE value is less than the set threshold, the training process will be ended, and C and parameter are output as the best parameters of the support vector machine algorithm model, thus completing the model training.

[0123] When the matching model reasoning is based on the support vector machine algorithm, the scene code is input, for example, the number of underground parking lot scene is 0010, which is converted into feature vector x=[0,0,1,0]. Then x is input into n SVM classifiers respectively, and the existence judgment of each target object is obtained, for example:

[0124] Classifier 1 (person): f1(x)=−1 → 0 (non-existent);

[0125] Classifier 2 (vehicle): f2(x)=1 → 1 (exists);

[0126] Classifier 3 (fire extinguisher): f3(x)=1 → 1 (exists);

[0127] …

[0128] Classifier n (trash can): fn(x) = −1 → 0 (does not exist).

[0129] Based on this, the target object that needs to be detected in this image can be determined.

[0130] In one embodiment, step S104 includes:

[0131] The prompt information and moving image are input into the YOLO-World model, and the YOLO-World model is used to perform multi-target detection on the target object. The detection results are then output as detection information.

[0132] In practical applications, the obtained scene information, detection area coordinate information, and target objects to be detected (such as vehicles, people, etc.) can be integrated into a set of Pormpts, which are then input into the YOLO-World model for multi-target detection, thereby outputting the corresponding detection results.

[0133] Specifically, such as Figure 3 As shown, the step of inputting the prompt information and moving image into the YOLO-World model, using the YOLO-World model to perform multi-target detection on the target object, and outputting the obtained detection results as detection information includes: steps S301 to S305.

[0134] Step S301: Use the Yolov8 network to extract features from the moving image to obtain the corresponding target feature map;

[0135] Step S302: Use the Transformer text encoder to segment the target object into words to obtain the corresponding word vectors;

[0136] Step S303: Interact and fuse the target feature map and the word vector to obtain the target feature vector;

[0137] Step S304: Perform predictive analysis on the target feature vector and output the corresponding detection box;

[0138] Step S305: Generate and output detection information based on the detection frame.

[0139] In this embodiment, the YOLO-World model consists of three parts: image feature extraction based on the YOLOv8 network, used to extract features from the input image; visual text alignment based on Transformer, used to vectorize the Prompt text; and multi-level cross-modal fusion of image and text based on Attention, used to fuse image features and text vectorization. Wherein:

[0140] The Yolov8 model structure has five stages, each of which is stacked by multiple C2f modules. The C2f module is a bottleneck structure, which first uses 1x1 convolution to reduce dimension, then uses 3x3 convolution to extract spatial features, and finally uses 1x1 convolution to increase dimension. The five stages are:

[0141] Stage1: Conv(3,64,k=3,s=2)→Conv(64,64,k=3,s=2)→C2f(64,64,n=1)→output feature map P1(160×160);

[0142] Stage2: C2f(64,128,n=2)→output feature map P2 (80×80);

[0143] Stage3: C2f(128,256,n=2)→output feature map P3 (40×40);

[0144] Stage4: C2f(256,512,n=2)→output feature map P4 (20×20);

[0145] Stage5: C2f(512,512,n=1)→SPPF(512,512)→output feature map P5 (10×10).

[0146] Where, Conv(3,64,k=3,s=2) represents that the input channel number is 3, the output channel is 64, the convolution kernel size is 3, the step is 2, and other Conv is the same. C2f(64,64,n=1) represents that the input channel number is 64, the output channel number is 64, and n represents the number of stacking is 1.

[0147] The backbone network finally outputs four scales of feature maps: P2 (80×80), P3 (40×40), P4 (20×20), and P5 (10×10). Then, the process of deep fusion is carried out through the Neck network:

[0148] The P5 feature map is reduced in dimension through the Conv module, and then up-sampled to the size of P4 (40×40). Then, the up-sampled P5 and P4 are added element by element. Then, the fused feature is processed through a C2f module to obtain a new P4_out. P4_out is up-sampled to the size of P3 (40×40) and fused with P3, and then processed through C2f to obtain P3_out.

[0149] P3_out is reduced in dimension by a Conv module and then down-sampled to the size of P4_out (20x20). The down-sampled P3_out is fused with P4_out. Then, the fused features are processed by a C2f module to obtain the final P4_out. Finally, P4_out is down-sampled to the size of P5 (10x10), fused with P5, and processed by a C2f to obtain the final P5_out.

[0150] Finally, three groups of feature maps of different sizes are output, which are used for detecting small targets (P3_out), medium targets (P4_out), and large targets (P5_out), respectively.

[0151] When the Prompt is text vectorized based on the Transformer, the target objects to be detected (such as people, vehicles, etc.) are encoded into high-dimensional semantic vectors. The implementation is to use a Transformer text encoder to tokenize the input target objects one by one. Then the word vectors of each target object are output.

[0152] When the Attention-based image-text multi-level cross-modal fusion technology is used to fuse image features and text vectors, the class word vectors (such as the embeddings of “person” and “vehicle”) output by the text encoder are used as PromptTokens. These PromptTokens interact with the visual feature maps, which can be achieved through the cross-attention mechanism, taking the visual features as Query and the text Prompt as Key and Value. The visual features at each spatial position are calculated to determine which text descriptions they are most relevant to. According to the attention weights, the visual features are weighted and enhanced to highlight the regions that match the text. The visual features processed by the adapter are used as the final output feature vectors.

[0153] Then, the output prediction is performed based on the obtained feature vectors. Specifically, a 1x1 convolution is first performed to adjust the number of channels to (n+4+1, where n represents the number of target objects, 4 represents x, y, w, and h, and 1 represents the confidence), and the output feature size is consistent with the input feature map size, with each pixel corresponding to a prediction anchor point, and each anchor point corresponding to n+5 information. After non-maximum suppression, the final detection box is output.

[0154] The detection box is then filtered. The coordinate position of the detection box needs to be located within the detection area to trigger an alarm, and the target box outside the detection area will not trigger an alarm. Therefore, the output detection box needs to be judged as follows:

[0155] ;

[0156] The coordinate range of the detection area is: x∈[x min ,x max], y∈[y min ,y max (i.e., the left boundary of the region is x) min The right boundary is x max The upper boundary is y min The lower boundary is y max If the parameters of the detection box are: center coordinates (x, y), width w, height h, then the left boundary of the detection box is x−w / 2, the right boundary is x+w / 2, the upper boundary is y−h / 2, and the lower boundary is y+h / 2.

[0157] The detection boxes that meet the above criteria are used as the final output detection boxes, and alarm information is generated and sent to relevant personnel.

[0158] Figure 4 This is a schematic block diagram of a positioning and movement inspection device 400 provided in an embodiment of the present invention. The device 400 includes:

[0159] The image acquisition unit 401 is used to acquire a fixed image through a fixed camera, and to acquire moving images of the same point at different angles through a moving camera;

[0160] The stitching and matching unit 402 is used to stitch together moving images from different angles to obtain a stitched image, and to perform scene matching between the stitched image and a fixed image to obtain corresponding scene information;

[0161] The first reasoning unit 403 is used to perform target object reasoning based on the scene information to obtain target object reasoning information containing the target object to be detected.

[0162] The second reasoning unit 404 is used to combine the scene information and the target object reasoning information into prompt information, and based on the prompt information, to perform model reasoning on the target object through a classification model to obtain the corresponding detection information.

[0163] In one embodiment, such as Figure 5 As shown, the splicing and matching unit 402 includes:

[0164] The standardization unit 501 is used to perform standardization processing on each of the moving images and extract image features from the standardized moving images.

[0165] The coordinate encoding unit 502 is used to perform coordinate encoding processing on the image features to obtain a first target feature with location information;

[0166] The context association unit 503 is used to perform context association on the target feature by alternately stacking self-attention and cross-attention to obtain a second target feature;

[0167] The similarity matching unit 504 is configured to calculate the similarity of the second target feature, and perform mutual nearest neighbor matching according to the calculation result of the similarity, so as to obtain an initial matching pair.

[0168] The image stitching unit 505 is configured to perform geometric constraint filtering on the initial matching pair, to obtain a reliable matching pair after filtering, and then stitch each of the moving images into the stitched image based on the reliable matching pair.

[0169] In an embodiment, the stitching matching unit 402 further includes:

[0170] The similarity calculation unit is configured to calculate the similarity of the stitched image and the fixed image by using a ResNet-18 network.

[0171] The image setting unit is configured to set the fixed image with the highest similarity score as a matching image.

[0172] The information summarizing unit is configured to extract corresponding scene and detection region coordinate information according to the matching image, and summarize the information as the scene information.

[0173] In an embodiment, the first inference unit 403 includes:

[0174] The scene encoding unit is configured to perform encoding processing on the scene information, to obtain corresponding scene encoding.

[0175] The classification prediction unit is configured to perform classification prediction on the scene encoding by using a support vector machine model, and set the result of the classification prediction as the target object inference information.

[0176] In an embodiment, the alignment mobile inspection device 400 includes:

[0177] The data set construction unit is configured to construct a training data set, wherein the training data set includes a plurality of training target objects and corresponding training scene data.

[0178] The framework building unit is configured to set a classifier in a one-to-one manner based on the training target objects, and set a radial basis function and a decision function, to build an initial support vector machine framework.

[0179] The model training unit is configured to train the initial support vector machine framework by using the training scene data, to obtain the support vector machine model.

[0180] In an embodiment, the second inference unit 404 includes:

[0181] The detection output unit is configured to input the prompt information and the moving image into a Yolo-World model, perform multi-target detection on the target object by using the Yolo-World model, and output a detection result obtained as detection information.

[0182] In an embodiment, as shown in Figure 6 The detection output unit includes:

[0183] The feature extraction unit 601 is configured to perform feature extraction on the moving image by using a Yolov8 network to obtain a corresponding target feature map.

[0184] The word segmentation unit 602 is configured to perform word segmentation processing on the target object by using a Transformer text encoder to obtain a corresponding word vector.

[0185] The interaction fusion unit 603 is configured to perform interaction fusion on the target feature map and the word vector to obtain a target feature vector.

[0186] The detection frame output unit 604 is configured to perform prediction analysis on the target feature vector and output a corresponding detection frame.

[0187] The information generation unit 605 is configured to generate detection information based on the detection frame and output the detection information.

[0188] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part are described in the description of the embodiments of the method part, and will not be described here.

[0189] The embodiments of the present application also provide a computer readable storage medium, which has a computer program stored thereon, and the computer program can implement the steps provided by the above embodiments when executed. The storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0190] The embodiments of the present application also provide a computer device, which can include a memory and a processor, the memory has a computer program stored therein, and the processor can implement the steps provided by the above embodiments when calling the computer program in the memory. Of course, the computer device can also include various network interfaces, power supplies and other components.

[0191] The various embodiments described in the specification are presented for purposes of illustration and description. Each of the embodiments highlight a different aspect of the application. The embodiments are not mutually exclusive, and can be combined in various manners. The embodiments disclosed herein are not exhaustive of the ways in which the application can be practiced. Numerous modifications and adaptations will be apparent to those skilled in the art. The embodiments disclosed herein are merely exemplary in nature and are not intended to limit the scope of the application.

[0192] It should also be noted that the terms "first", "second", and the like, do not denote any order, quantity, combination, or importance, but rather are used to identify one element from another. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. A method for alignment and movement inspection, characterized in that, include: A fixed image is obtained by using a fixed camera, and a moving image is obtained from the same point at different angles by using a moving camera; Moving images from different angles are stitched together to obtain a stitched image, and the stitched image is matched with a fixed image to obtain the corresponding scene information; Based on the scene information, target object reasoning is performed to obtain target object reasoning information containing the target object to be detected; The scene information and the target object reasoning information are combined into prompt information, and based on the prompt information, the target object is inferred by a classification model to obtain the corresponding detection information; The step of performing target object reasoning based on the scene information to obtain target object reasoning information containing the target object to be detected includes: The scene information is encoded to obtain the corresponding scene code; The scene code is classified and predicted using a support vector machine model, and the classification and prediction results are set as the inference information for the target object.

2. The alignment and movement inspection method according to claim 1, characterized in that, The process involves stitching together moving images from different angles to obtain a stitched image, and then matching the stitched image with a fixed image to obtain corresponding scene information, including: Each of the moving images is standardized, and image features are extracted from the standardized moving images. The image features are subjected to coordinate encoding to obtain a first target feature with location information; By alternately stacking self-attention and cross-attention, the target features are context-associated to obtain the second target features; Calculate the similarity of the second target feature, and perform nearest neighbor matching based on the similarity calculation results to filter out initial matching pairs; Geometric constraint filtering is applied to the initial matching pairs to obtain filtered reliable matching pairs. Then, based on the reliable matching pairs, the moving images are stitched together to form the stitched image.

3. The alignment and movement inspection method according to claim 1, characterized in that, The process of stitching together moving images from different angles to obtain a stitched image, and then matching the stitched image with a fixed image to obtain corresponding scene information, further includes: A ResNet-18 network is used to calculate the similarity between the stitched image and the fixed image; The fixed image with the highest similarity score is used as the matching image; The corresponding scene and detection area coordinates are extracted from the matched image and summarized into the scene information.

4. The alignment and movement inspection method according to claim 1, characterized in that, The construction process of the support vector machine model includes: Construct a training dataset, wherein the training dataset includes multiple training targets and corresponding training scene data; Based on the training target, a classifier is set up in a one-to-one manner, and radial basis functions and decision functions are set up to build an initial support vector machine framework; The initial support vector machine framework is trained using the training scenario data to obtain the support vector machine model.

5. The alignment and movement inspection method according to claim 1, characterized in that, The step of combining the scene information and target object reasoning information into prompt information, and based on the prompt information, performing model reasoning on the target object through a classification model to obtain corresponding detection information, includes: The prompt information and moving image are input into the YOLO-World model, and the YOLO-World model is used to perform multi-target detection on the target object. The detection results are then output as detection information.

6. The alignment and movement inspection method according to claim 5, characterized in that, The step of inputting the prompt information and the moving image into the YOLO-World model, using the YOLO-World model to perform multi-target detection on the target object, and outputting the detection results as detection information includes: The moving image is used to extract features to obtain the corresponding target feature map; The target object is segmented using the Transformer text encoder to obtain the corresponding word vectors; The target feature map and the word vector are interactively fused to obtain the target feature vector; The target feature vector is predicted and analyzed, and the corresponding detection box is output. Detection information is generated and output based on the detection frame.

7. A positioning and moving inspection device, characterized in that, include: The image acquisition unit is used to acquire a fixed image through a fixed camera, and to acquire moving images of the same point at different angles through a moving camera; The stitching and matching unit is used to stitch moving images from different angles to obtain a stitched image, and to perform scene matching between the stitched image and a fixed image to obtain corresponding scene information; The first reasoning unit is used to perform target object reasoning based on the scene information to obtain target object reasoning information containing the target object to be detected. The second reasoning unit is used to combine the scene information and the target object reasoning information into prompt information, and based on the prompt information, to perform model reasoning on the target object through a classification model to obtain the corresponding detection information; The first inference unit includes: A scene encoding unit is used to encode the scene information to obtain the corresponding scene code; The classification prediction unit is used to perform classification prediction on the scene code using a support vector machine model, and set the classification prediction result as the inference information of the target object.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the alignment movement inspection method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the alignment movement inspection method as described in any one of claims 1 to 6.

Images