A vision-based lost property retrieval method and system along railway lines

Abstract

The present invention belongs to the field of electrical data processing, and provides a lost object retrieval method and system along a railway based on vision, comprising acquiring a monitoring video along the railway and performing frame processing to obtain a search library, filtering all images and lost object images in the search library to obtain a filtered image search library and a lost object filtered image; based on the filtered image search library and the lost object filtered image, using a trained double-branch convolutional neural network model to perform feature extraction to obtain a search feature library and lost object features; by calculating the difference value index and fluctuation value index of all features in the search feature library and the lost object features, respectively extracting the binary search library image features of the search feature library and the binary lost object image features of the lost object features, calculating the matching degree of the binary search library image features and the binary lost object image features, and outputting the retrieval result according to the matching degree; the disadvantages of the lost object retrieval method along a railway based on vision are solved.

Description

Technical Field

[0001] The invention belongs to the technical field of electronic data processing, and in particular relates to a vision-based lost property retrieval method and system along a railway. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Railway transportation occupies an extremely important position in the modern logistics and passenger transportation system. Its network covers a wide area, covering various terrains and complex environments. In the process of railway operation, passengers or goods often lose items along the railway. These lost items may not only include important personal belongings of passengers, affecting their travel experience and convenience of life, but also the loss of some goods may pose a potential threat to the safety of railway operations and the integrity of cargo transportation. Timely discovery and handling of lost items can effectively ensure the safety and order of railway transportation and reduce the risk of accidents caused by foreign objects.

[0004] The existing lost property retrieval methods along the railway are mainly divided into two categories: manual inspection-based technology and vision-based technology. The manual inspection-based technology mainly relies on railway staff to walk along the railway line or use vehicles for regular inspections, and rely on naked eye observation to find lost objects. This method is extremely inefficient and requires a lot of manpower, material resources and time costs. Especially when the railway line is long and the geographical environment is complex, it is difficult to achieve a comprehensive and timely search for lost objects. The vision-based detection technology captures video images along the railway by installing cameras, and then uses image processing technology to find lost objects. However, the environment along the railway is complex and diverse, and is affected by various factors such as lighting and occlusion. There are still the following problems: First, the environment along the railway is complex and diverse, including background elements such as rails, sleepers, bridges, tunnels, different types of vegetation, buildings, and various railway facilities. There may be similar visual features between the background and the lost objects, which can easily cause false detection. Second, there are many types of lost objects along the railway, with different shapes, sizes, and colors. The frequency of occurrence of different types of lost objects varies greatly. The imbalance of data samples may lead to poor detection performance of the model for some low-frequency lost objects. Third, there are usually a large number of surveillance cameras along the railway, which generate massive amounts of video data. Existing algorithms are difficult to meet real-time requirements when processing large-scale data.

[0005] However, the existing lost property retrieval methods along railways adopt a processing path with a single-branch network structure. Faced with complex data along the railway, they often focus on the more obvious overall features and find it difficult to fully extract features of multiple levels and types. In addition, the environment along the railway is complex, and image acquisition is easily affected by electromagnetic interference, weather factors, etc., which produce noise. These noises will reduce image clarity and contrast, and interfere with the accurate extraction of features. Moreover, high-dimensional features used for lost property retrieval will reduce retrieval efficiency and fail to meet real-time requirements. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a vision-based lost property retrieval method and system along the railway. The present invention first proposes to filter the original image, fuse the features of the original image and the filtered image, and improve the image description ability; then it proposes to use two branches of CNN to learn the weight values ​​of the original image and the filtered image respectively, and the high-dimensional features of the two branches complement each other, so that the deep spatial features are significantly enhanced; finally, it is proposed to binarize the features and calculate the urban distance to save retrieval time.

[0007] According to some embodiments, the first solution of the present invention provides a vision-based lost property retrieval method along a railway, using the following technical solution:

[0008] A vision-based lost property retrieval method along a railway, comprising:

[0009] Acquire surveillance videos along the railway and perform frame processing to obtain a search library, perform filtering processing on all images and lost object images in the search library to obtain a filtered image search library and a lost object filtered image;

[0010] Based on the filtered image search library and the lost object filtered image, the trained dual-branch convolutional neural network model is used to extract features to obtain the search feature library and lost object features;

[0011] By calculating the difference value index and fluctuation value index of all features in the search feature library and the lost object features, binarization processing is performed to extract the binary search library image features of the search feature library and the binary lost object image features of the lost object features, respectively, as follows:

[0012] Calculate the difference value index and fluctuation value index of all features and lost object features in the search feature library respectively;

[0013] Construct a search feature threshold based on the difference value index and fluctuation value index of all features in the search feature library;

[0014] Construct the lost object feature threshold according to the difference value index and fluctuation value index of the lost object feature;

[0015] Binarization is performed on all features in the search feature library and lost object features based on the search feature threshold and the lost object feature threshold respectively;

[0016] Obtaining binary search library image features and binary lost object image features;

[0017] Calculate the matching degree between the binary search library image features and the binary lost object image features, and output the retrieval results based on the matching degree.

[0018] Furthermore, all images in the search library and the lost object images are filtered to obtain a filtered image search library and a lost object filtered image, specifically:

[0019] Perform non-local mean filtering on all images in the search library and lost object images, and obtain corresponding filtering results as the search library guide and lost object guide;

[0020] Based on the search library guide and the lost object guide, all images in the search library and the lost object images are guided to complete the guided filtering, so as to obtain the filtered image search library and the lost object filtered images.

[0021] Furthermore, based on the filter image search library, the trained dual-branch convolutional neural network model is used to extract features, specifically:

[0022] Traverse each image in the search library and the corresponding filtered image in the filter image search library;

[0023] Input into the trained dual-branch convolutional neural network model for feature extraction;

[0024] Get the search feature library.

[0025] Furthermore, based on the lost object filter image, the trained dual-branch convolutional neural network model is used to extract features, specifically:

[0026] Input the lost object image and the lost object filtered image into the trained dual-branch convolutional neural network model for feature extraction;

[0027] Get the lost item feature.

[0028] Furthermore, the dual-branch convolutional neural network model is specifically:

[0029] The dual-branch convolutional neural network model includes two convolutional neural network model structure branches, and the two convolutional neural network model structure branches share parameters;

[0030] Each convolutional neural network model structure branch contains 4 stacked convolutional neural networks. Each stacked convolutional neural network contains 1 convolution layer and 1 maximum pooling layer. After convolution and pooling, features are extracted through 1 fully connected layer.

[0031] The features of the two convolutional neural network model structure branches are merged at the end of the branch through another new fully connected layer to obtain the final feature representation;

[0032] The final feature representation is input into the SVM classifier to complete the classification task;

[0033] Among them, the outputs of the convolutional layer and the fully connected layer in the dual-branch convolutional neural network model are activated by an activation function, and a regularization layer is added to avoid overfitting.

[0034] Furthermore, the extraction of the binary search library image features of the search feature library and the binary lost object image features of the lost object features is specifically:

[0035] Characteristics of lost items and search feature library Binarization is performed separately, as shown in the following two formulas:

[0036] ;

[0037] ;

[0038] Among them, the lost object image , represents the characteristic component, represents the feature dimension, express Kuzhongdi The features of the image, To find the number of images in the library; Binarize the missing object image features and binarization to find library image features , The first The value of the dimension, The first The value of the dimension;

[0039] It means to find the binary features of the sth image in the library, as shown below:

[0040] .

[0041] Furthermore, the calculation of the matching degree between the binary search library image features and the binary lost object image features, and outputting the search results according to the matching degree, is specifically as follows:

[0042] Calculate the matching degree of the binary search library image features and the binary lost object image features corresponding to each image in the search library respectively;

[0043] The calculated matching degrees are sorted from small to large, and the image in the search library corresponding to the largest matching degree is used as the retrieval result.

[0044] According to some embodiments, the second solution of the present invention provides a vision-based lost property retrieval system along the railway, which adopts the following technical solution:

[0045] A vision-based lost property retrieval system along a railway, comprising:

[0046] The image filtering processing module is used to obtain the monitoring video along the railway and perform frame processing to obtain a search library, and filter all images and lost object images in the search library to obtain a filtered image search library and a lost object filtered image;

[0047] An image feature extraction module is used to extract features based on the filtered image search library and the lost object filtered image using a trained dual-branch convolutional neural network model to obtain a search feature library and lost object features;

[0048] The image binarization matching module is used to perform binarization processing by calculating the difference value index and fluctuation value index of all features in the search feature library and the lost object features, and respectively extract the binary search library image features of the search feature library and the binary lost object image features of the lost object features, specifically:

[0049] Calculate the difference value index and fluctuation value index of all features and lost object features in the search feature library respectively;

[0050] Construct a search feature threshold based on the difference value index and fluctuation value index of all features in the search feature library;

[0051] Construct the lost object feature threshold according to the difference value index and fluctuation value index of the lost object feature;

[0052] Binarization is performed on all features in the search feature library and lost object features based on the search feature threshold and the lost object feature threshold respectively;

[0053] Obtaining binary search library image features and binary lost object image features;

[0054] Calculate the matching degree between the binary search library image features and the binary lost object image features, and output the retrieval results based on the matching degree.

[0055] According to some embodiments, a third aspect of the present invention provides a computer-readable storage medium.

[0056] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a vision-based lost property retrieval method along a railway as described in the first aspect above.

[0057] According to some embodiments, a fourth aspect of the present invention provides a computer device.

[0058] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a vision-based lost property retrieval method along a railway as described in the first aspect above are implemented.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] The present invention proposes a new binarization method. In the image comparison stage, after obtaining the binary lost object image features and the binary search library image features, they are first binarized respectively, then the matching degree is calculated, and finally the retrieval result is output according to the matching degree; the feature binarization process is based on the features to be binarized, and the difference value index and the fluctuation value index are calculated respectively, and then the feature threshold is constructed according to the difference value index and the fluctuation value index, and finally the feature is binarized according to the threshold. The feature binarization method effectively utilizes the information of the feature itself, and effectively improves the retrieval efficiency without losing or losing less accuracy.

[0061] This paper proposes a new vision-based lost property retrieval method along the railway to solve the problems of weak image description ability along the railway, difficulty in extracting multi-level features by a single-branch CNN network, and retrieval efficiency that does not meet the real-time requirements. First, it is proposed to filter the original image, fuse the features of the original image and the filtered image, and improve the image description ability. Then, it is proposed to use two branches of CNN to learn the weight values ​​of the original image and the filtered image respectively, and the high-dimensional features of the two branches complement each other, so that the deep spatial features are significantly enhanced. Finally, it is proposed to binarize the features and calculate the distance between urban areas to save retrieval time. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0063] Figure 1 is a flow chart of a method for retrieving lost items along a railway based on vision in an embodiment of the present invention;

[0064] Figure 2 is a flowchart of image filtering processing in an embodiment of the present invention;

[0065] Figure 3 is a structural diagram of a dual-branch convolutional neural network model in an embodiment of the present invention;

[0066] Figure 4 is a flowchart of image binarization matching in an embodiment of the present invention;

[0067] Figure 5 It is a flowchart of image binarization processing in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0069] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0070] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0071] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0072] Embodiment 1

[0073] like Figure 1 As shown, this embodiment provides a vision-based lost property retrieval method along the railway. This embodiment uses the method applied to a server as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a terminal, a server, and a system, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0074] Acquire surveillance videos along the railway and perform frame processing to obtain a search library, perform filtering processing on all images and lost object images in the search library to obtain a filtered image search library and a lost object filtered image;

[0075] Based on the filtered image search library and the lost object filtered image, the trained dual-branch convolutional neural network model is used to extract features to obtain the search feature library and lost object features;

[0076] Binarization is performed by calculating the difference value index and fluctuation value index of all features in the search feature library and the lost object features, and the binary search library image features of the search feature library and the binary lost object image features of the lost object features are extracted respectively. The matching degree of the binary search library image features and the binary lost object image features is calculated, and the retrieval results are output according to the matching degree.

[0077] like Figure 1 As shown, the basic framework of the vision-based lost property retrieval method along the railway proposed in the present invention includes three main steps: filtering, dual-branch CNN network feature extraction, and feature comparison.

[0078] Recording lost item images of suspected lost items The surveillance videos along the railway line collected by surveillance shooting are , first of all Perform framing and obtain the search library , filter all images and lost items in the search library to obtain the filtered image search library and the missing object filter image ; Then use the dual-branch CNN network to extract the filter image search library and the missing object filter image The characteristics of , , as shown in formulas (1) and (2):

[0079] (1);

[0080] (2);

[0081] in, represents the characteristic component, represents the feature dimension, Indicates the search for the first The features of the image, To find the number of images in the library.

[0082] Secondly, the characteristics of the lost items and search feature library Binarize them respectively to obtain the binary lost object image features and binarization to find library image features .

[0083] Finally, based on the feature and , compare the matching degree between the lost object image and the search library image to determine the retrieval result.

[0084] The detailed process is as follows:

[0085] First, all images in the search library and lost object images are filtered to obtain a filtered image search library and a lost object filtered image, specifically:

[0086] Perform non-local mean filtering on all images in the search library and lost object images, and obtain corresponding filtering results as the search library guide and lost object guide;

[0087] Based on the search library guide and the lost object guide, all images in the search library and the lost object images are guided to complete the guided filtering, so as to obtain the filtered image search library and the lost object filtered images.

[0088] Considering the complex environment along the railway, image acquisition is susceptible to electromagnetic interference, weather factors, etc., which can generate noise. These noises will reduce the image clarity and contrast, and interfere with the accurate extraction of lost object features. At the same time, filtering can improve image quality and stability, make object edges clearer and more coherent, enhance visual effects, and make image quality more in line with subsequent search expectations, thereby improving the accuracy and robustness of the algorithm. This embodiment proposes a filtering method for images along the railway, such as Figure 2 As shown, firstly, the input noisy image Perform non-local mean filtering and obtain the filtering result as a guide , then based on the boot Guide the input image to complete the guided filtering and obtain the denoised image .

[0089] Assume the input image is , use formula (3) to grayscale it to obtain the grayscale noise image as follows:

[0090] (3);

[0091] in, , , The noisy images are The pixel values ​​of the R, G, and B channels.

[0092] , Represents the pixel value of the point. Indicates that the center pixel is And the size is The present invention is based on the search area of ​​grayscale noise image The guide G is obtained by performing non-local mean filtering as follows:

[0093] First, for the pixel and search area Each pixel in , respectively obtain the center point and the size Image blocks , , then calculate and The distance between As shown in formula (4):

[0094] (4);

[0095] Finally, calculate the pixel The corresponding value in the filtered image As shown in formula (5):

[0096] (5);

[0097] Bootstrap based Guided Gray Noise Image Complete guided filtering to obtain denoised image ,as follows:

[0098] First define the filter window The size is , solve for the filter coefficient As shown in formula (6) and formula (7):

[0099] (6);

[0100] (7);

[0101] in, Indicates the guide The sum of all pixels in the filter window, For the guide The mean value of all pixels in the filter window, For the guide The variance of all pixels in the filter window, Gray noise image The mean value over the filter window.

[0102] Then calculate the grayscale noise image The filtering result is shown in formula (8):

[0103] (8);

[0104] Here, Represents the index values ​​of different positions in the filter window.

[0105] Finally, calculate the noisy image The filtering result is shown in formula (9):

[0106] (9);

[0107] in, , , They are color images and denoised images respectively. The pixel values ​​of the R, G, and B channels.

[0108] After confirming the filtering method of this embodiment, in the lost object retrieval task along the railway, traverse the search library Each image in , use the above filtering method to filter it and obtain the filtering result , forming a filter image search library When a user loses an item, get the lost item image , filter it and get the lost object filtered image .

[0109] Based on the filter image search library, the trained dual-branch convolutional neural network model is used for feature extraction, specifically:

[0110] Traverse each image in the search library and the corresponding filtered image in the filter image search library;

[0111] Input into the trained dual-branch convolutional neural network model for feature extraction;

[0112] Get the search feature library.

[0113] Based on the lost object filter image, the trained dual-branch convolutional neural network model is used to extract features, specifically:

[0114] Input the lost object image and the lost object filtered image into the trained dual-branch convolutional neural network model for feature extraction;

[0115] Get the lost item feature.

[0116] The dual-branch convolutional neural network model is specifically:

[0117] The dual-branch convolutional neural network model includes two convolutional neural network model structure branches, and the two convolutional neural network model structure branches share parameters;

[0118] Each convolutional neural network model structure branch contains 4 stacked convolutional neural networks. Each stacked convolutional neural network contains 1 convolution layer and 1 maximum pooling layer. After convolution and pooling, features are extracted through 1 fully connected layer.

[0119] The features of the two convolutional neural network model structure branches are merged at the end of the branch through another new fully connected layer to obtain the final feature representation;

[0120] The final feature representation is input into the SVM classifier to complete the classification task;

[0121] Among them, the outputs of the convolutional layer and the fully connected layer in the dual-branch convolutional neural network model are activated by an activation function, and a regularization layer is added to avoid overfitting.

[0122] In the vision-based lost property retrieval along the railway, the environment along the railway is complex and changeable, and the original images collected are Contains rich but messy information, such as various natural scenery, track facilities, etc. Filtered image To a certain extent, it has been optimized to remove some noise interference and make the edges of objects more prominent, but the original image It retains the most realistic scene details and can provide the original characteristic distribution of lost objects in complex backgrounds.

[0123] This embodiment proposes a Figure 3The dual-branch convolutional neural network model (CNN) shown in the figure, i.e., the dual-branch CNN model, achieves high-dimensional feature complementation by learning the two different images, the comprehensive detail features provided by the original image and the precise prominent features provided by the filtered image, so as to more accurately identify lost items along the railway. The dual-branch convolutional neural network model includes two CNN structure branches, and the two CNN structure branches share parameters. Each CNN structure branch includes four stacked convolutional neural networks ConvNet, and each ConvNet includes one convolution layer and one maximum pooling layer (Max-Pooling layer). After convolution and pooling, features are extracted through one fully connected layer. The features of the two CNN structure branches are merged through a new fully connected layer at the end of the branch to obtain the final feature representation. The final feature representation is then input into the SVM classifier to complete the classification task. The outputs of the convolutional layer and the fully connected layer in the network are activated by the activation function (ReLU), which shortens the network training time. At the same time, a regularization (Dropout) layer is added to avoid overfitting. Table 1 shows the key layer parameters of the network.

[0124] Table 1. Key layer parameters of the dual-branch CNN structure for railway image feature extraction proposed in this paper

[0125]

[0126] This embodiment is mainly aimed at the vision-based lost property retrieval task along the railway. In this task, the proposed dual-branch CNN model for railway image feature extraction is mainly used for image feature extraction. In order to ensure the accuracy of the image description, the model needs to be trained before the image retrieval task is carried out to obtain the best model parameters. In the model training stage, with the help of professional image acquisition equipment, all-round and multi-angle shooting is carried out along the railway track, covering various scenes such as different geographical areas, weather conditions and various states of railway facilities, so as to obtain rich and representative image data, and then the collected images are annotated and sorted by category, and finally the annotated data is input into the model for training. In the feature extraction stage, the model first loads the optimal weight parameters that have been trained, and then derives the dual branches based on the output original image and filtered image, and extracts the features of the last fully connected layer as image features.

[0127] In the lost property retrieval task along the railway, traverse the search library Each image in and the corresponding filtered image , and input it into the above two-branch CNN model to extract features , forming a search feature library :

[0128]

[0129] in, .

[0130] The lost item image and its filtered image Input the above two-branch CNN model to extract the features of lost objects ,in, represents the characteristic component, represents the feature dimension, Indicates the search for the first The features of the image, To find the number of images in the library.

[0131] In a specific embodiment, the binary processing is performed by calculating the difference value index and the fluctuation value index of all features and lost object features in the search feature library, specifically:

[0132] Calculate the difference value index and fluctuation value index of all features and lost object features in the search feature library respectively;

[0133] Construct a search feature threshold based on the difference value index and fluctuation value index of all features in the search feature library;

[0134] Construct the lost object feature threshold according to the difference value index and fluctuation value index of the lost object feature;

[0135] Binarization is performed on all features in the search feature library and lost object features based on the search feature threshold and the lost object feature threshold respectively;

[0136] Obtain the binary search library image features and the binary lost object image features.

[0137] The calculation of the matching degree between the binary search library image features and the binary lost object image features, and outputting the search results according to the matching degree, is specifically as follows:

[0138] Calculate the matching degree of the binary search library image features and the binary lost object image features corresponding to each image in the search library respectively;

[0139] The calculated matching degrees are sorted from small to large, and the image in the search library corresponding to the largest matching degree is used as the retrieval result.

[0140] like Figure 4As shown in the figure, when a computer performs numerical calculations, it needs to operate on each digit. For numbers with many digits after the decimal point, the computer needs to process more digital information, which usually takes more time. In the image comparison stage, after obtaining the image features of the lost object and searching the feature library, they are first binarized, then the matching degree is calculated, and finally the retrieval results are output according to the matching degree.

[0141] like Figure 5 As shown in the figure, firstly, based on the features to be binarized, the difference value index and the fluctuation value index are calculated respectively, then the feature threshold is constructed according to the difference value index and the fluctuation value index, and finally the feature is binarized according to the threshold. This feature binarization method effectively utilizes the information of the feature itself and improves the comparison efficiency without losing or losing less accuracy. The specific process is as follows:

[0142] Assume that the feature to be binarized is , first calculate its difference value index as shown in formula (10):

[0143] (10);

[0144] in, , are the maximum and minimum values ​​of the characteristic components, represents the characteristic component, Represents the feature dimension.

[0145] Then calculate the volatility index as shown in formula (11):

[0146] (11);

[0147] Secondly, based on the difference value and fluctuation value Construct a binarization threshold as shown in formula (12).

[0148] (12);

[0149] Finally, for the features The binarization is performed as shown in formula (13):

[0150] (13);

[0151] in, Represents binary features No. The value of the dimension.

[0152] Using the above feature binarization method, the lost object features and search feature library Binarization is performed respectively, and the results are shown in formulas (14) and (15):

[0153] (14);

[0154] (15);

[0155] Among them, the binary lost object image features and binarization to find library image features , It represents the binary features of the sth image in the search library, as shown in formula (16):

[0156] (16);

[0157] Computing the missing object image and find images in the library The matching degree is shown in formula (17):

[0158] (17);

[0159] in, is the index to find the image in the library, Indicates searching for the sth image in the library. The first The value of the dimension, The first The value of the dimension.

[0160] based on The comprehensive matching degree is constructed as shown in formula (18):

[0161] (18);

[0162] right Sort from small to large and return the search result as shown in formula (19):

[0163] (19);

[0164] in, Indicates that the function is obtained When the maximum value is obtained, find the corresponding image in the library.

[0165] This embodiment proposes a new vision-based lost property retrieval method along the railway, and the key problems to be solved are: 1. The image description ability along the railway is weak. 2. It is difficult for a single-branch CNN network to extract multi-level features. 3. The retrieval efficiency does not meet the real-time requirements. The present invention first proposes to filter the original image, fuse the features of the original image and the filtered image, and improve the image description ability. Then it is proposed to use two branches of CNN to learn the weight values ​​of the original image and the filtered image respectively, and the high-dimensional features of the two branches complement each other, so that the deep spatial features are significantly enhanced. Finally, it is proposed to binarize the features and calculate the distance between urban areas to save retrieval time.

[0166] This embodiment proposes an amplification method for railway images: a filtering method for railway images is proposed, the original image is filtered, the features of the original image and the filtered image are integrated, and the image description capability is improved.

[0167] This embodiment proposes a new dual-branch CNN structure to extract multi-level features of railway images: it proposes to use two branch CNNs to learn the weight values ​​of the original image and the filtered image respectively, and the high-dimensional features of the dual branches complement each other, so that the deep spatial features are significantly enhanced.

[0168] This embodiment proposes a new feature binarization method: a new binarization method is proposed to binarize the extracted railway image features, which effectively improves the retrieval efficiency without losing or losing less accuracy.

[0169] Embodiment 2

[0170] This embodiment provides a vision-based lost property retrieval system along a railway, including:

[0171] The image filtering processing module is used to obtain the monitoring video along the railway and perform frame processing to obtain a search library, and filter all images and lost object images in the search library to obtain a filtered image search library and a lost object filtered image;

[0172] An image feature extraction module is used to extract features based on the filtered image search library and the lost object filtered image using a trained dual-branch convolutional neural network model to obtain a search feature library and lost object features;

[0173] The image binarization matching module is used to perform binarization processing by calculating the difference value index and fluctuation value index of all features in the search feature library and the lost object features, and respectively extract the binary search library image features of the search feature library and the binary lost object image features of the lost object features, specifically:

[0174] Calculate the difference value index and fluctuation value index of all features and lost object features in the search feature library respectively;

[0175] Construct a search feature threshold based on the difference value index and fluctuation value index of all features in the search feature library;

[0176] Construct the lost object feature threshold according to the difference value index and fluctuation value index of the lost object feature;

[0177] Binarization is performed on all features in the search feature library and lost object features based on the search feature threshold and the lost object feature threshold respectively;

[0178] Obtaining binary search library image features and binary lost object image features;

[0179] Calculate the matching degree between the binary search library image features and the binary lost object image features, and output the retrieval results based on the matching degree.

[0180] The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0181] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0182] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0183] Embodiment 3

[0184] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the vision-based lost property retrieval method along a railway as described in the first embodiment above are implemented.

[0185] Embodiment 4

[0186] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of a vision-based lost property retrieval method along a railway as described in the first embodiment above are implemented.

[0187] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0188] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0190] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0191] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0192] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A vision-based lost property retrieval method along a railway, characterized in that: include: Acquire surveillance videos along the railway and perform frame processing to obtain a search library, filter all images and lost object images in the search library to obtain a filtered image search library and lost object filtered images; Based on the filtered images in the filter image search library and all the images in the search library, the lost object filtered images and the lost object images, the trained two-branch convolutional neural network model is used to extract features to obtain the search feature library and the lost object features; By calculating the difference value index and fluctuation value index of all features in the search feature library and the lost object features, binarization processing is performed to extract the binary search library image features of the search feature library and the binary lost object image features of the lost object features, respectively, as follows: The difference value index and fluctuation value index of all features and lost object features in the search feature library are calculated respectively. The calculation principle is as follows: Assume that the feature to be binarized is , the difference value index is calculated as follows: ； in, , are the maximum and minimum values ​​of the characteristic components, represents the characteristic component, represents the feature dimension, Represents binary features No. The value of the dimension; The calculation of the volatility indicator is as follows: ； Construct a search feature threshold based on the ratio of the difference value index and the fluctuation value index of all features in the search feature library; The lost object feature threshold is constructed according to the ratio of the difference value index and the fluctuation value index of the lost object feature; Binarization is performed on all features in the search feature library and lost object features based on the search feature threshold and the lost object feature threshold respectively; Obtaining binary search library image features and binary lost object image features; Calculate the matching degree between the binary search library image features and the binary lost object image features, and output the retrieval results based on the matching degree.

2. A method for retrieving lost items along a railway based on vision as claimed in claim 1, characterized in that: All images in the search library and lost object images are filtered to obtain a filtered image search library and a lost object filtered image, specifically: Perform non-local mean filtering on all images in the search library and lost object images, and obtain corresponding filtering results as the search library guide and lost object guide; Based on the search library guide and the lost object guide, all images in the search library and the lost object images are guided to complete the guided filtering, so as to obtain the filtered image search library and the lost object filtered images.

3. A method for retrieving lost items along a railway based on vision as claimed in claim 1, characterized in that: Based on the filtered images in the filter image search library and all the images in the search library, feature extraction is performed using the trained dual-branch convolutional neural network model, specifically: Traverse each image in the search library and the corresponding filtered image in the filter image search library; Input into the trained dual-branch convolutional neural network model for feature extraction; Get the search feature library.

4. A method for retrieving lost objects along a railway based on vision as claimed in claim 1, characterized in that: Based on the lost object filter image and the lost object image, the trained dual-branch convolutional neural network model is used to extract features, specifically: Input the lost object image and the lost object filtered image into the trained dual-branch convolutional neural network model for feature extraction; Get the lost item feature.

5. A method for retrieving lost items along a railway based on vision as claimed in claim 1, characterized in that: The dual-branch convolutional neural network model is specifically: The dual-branch convolutional neural network model includes two convolutional neural network model structure branches, and the two convolutional neural network model structure branches share parameters; Each convolutional neural network model structure branch contains 4 stacked convolutional neural networks. Each stacked convolutional neural network contains 1 convolution layer and 1 maximum pooling layer. After convolution and pooling, features are extracted through 1 fully connected layer. The features of the two convolutional neural network model structure branches are merged at the end of the branch through another new fully connected layer to obtain the final feature representation; The final feature representation is input into the SVM classifier to complete the classification task; Among them, the outputs of the convolutional layer and the fully connected layer in the dual-branch convolutional neural network model are activated by an activation function, and a regularization layer is added to avoid overfitting.

6. A method for retrieving lost items along a railway based on vision as claimed in claim 1, characterized in that: The binary search library image features of the extracted search feature library and the binary lost object image features of the lost object features are specifically: Characteristics of lost items and search feature library Binarization is performed separately, as shown in the following two formulas: ； ； Among them, the lost object image , represents the characteristic component, represents the feature dimension, express Kuzhongdi The features of the image, To find the number of images in the library; Binarize the missing object image features and binarization to find library image features , The first The value of the dimension, The first The value of the dimension; It means to find the binary features of the sth image in the library, as shown below: 。 7. A method for retrieving lost items along a railway based on vision as claimed in claim 1, characterized in that: Calculate the matching degree between the binary search library image features and the binary lost object image features, and output the search results according to the matching degree, specifically: Calculate the matching degree of the binary search library image features and the binary lost object image features corresponding to each image in the search library respectively; The calculated matching degrees are sorted from small to large, and the image in the search library corresponding to the largest matching degree is used as the retrieval result.

8. A vision-based lost property retrieval system along the railway, characterized in that: include: The image filtering processing module is used to obtain the monitoring video along the railway and perform frame processing to obtain a search library, and filter all images and lost object images in the search library to obtain a filtered image search library and a lost object filtered image; An image feature extraction module is used to extract features based on the filtered images in the filtered image search library and all images in the search library, the lost object filtered images and the lost object images, respectively, using the trained dual-branch convolutional neural network model to obtain a search feature library and lost object features; The image binarization matching module is used to perform binarization processing by calculating the difference value index and fluctuation value index of all features in the search feature library and the lost object features, and respectively extract the binary search library image features of the search feature library and the binary lost object image features of the lost object features, specifically: The difference value index and fluctuation value index of all features and lost object features in the search feature library are calculated respectively. The calculation principle is as follows: Assume that the feature to be binarized is , the difference value index is calculated as follows: ； in, , are the maximum and minimum values ​​of the characteristic components, represents the characteristic component, represents the feature dimension, Represents binary features No. The value of the dimension; The calculation of the volatility indicator is as follows: ； The search feature threshold is constructed according to the ratio of the difference value index and the fluctuation value index of all features in the search feature library; The lost object feature threshold is constructed according to the ratio of the difference value index and the fluctuation value index of the lost object feature; Binarization is performed on all features in the search feature library and lost object features based on the search feature threshold and the lost object feature threshold respectively; Obtaining binary search library image features and binary lost object image features; Calculate the matching degree between the binary search library image features and the binary lost object image features, and output the retrieval results based on the matching degree.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of a vision-based lost property retrieval method along a railway as described in any one of claims 1-7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a vision-based lost property retrieval method along a railway as described in any one of claims 1-7 are implemented.

Images