Container detection method, device, electronic device and storage medium

Abstract

An embodiment of the present invention provides a container detection method, which obtains a sequence of feature matrices to be processed, including feature matrices to be processed arranged according to a time dimension, each of which includes image feature values ​​of multiple containers to be detected arranged according to an adjacency relationship; performs a first feature extraction process on each frame of the feature matrix to be processed in the sequence to be processed to obtain a spatial feature vector of the sequence to be processed; performs a second feature extraction process on the spatial feature vector to obtain a spatiotemporal feature vector of the sequence to be processed; and determines detection results for multiple containers to be detected based on the spatiotemporal feature vector. By extracting the spatiotemporal feature vector from the spatial feature vector, the container can be automatically detected in the spatiotemporal dimension, thereby improving the efficiency of container detection and increasing the safety of container detection by eliminating the need for manual inspection.

Description

Technical Field

[0001] The present invention relates to the field of smart port technology, and in particular to a container detection method, device, electronic equipment and storage medium. Background Art

[0002] When loading containers onto a ship, they need to be secured on board. This is typically done by lashing the containers, which are then secured by personnel on a lashing bridge. Since loose container securing can affect the ship's safety during navigation, strict inspections are required. The existing inspection method involves manual inspections by inspectors on a lashing bridge after loading. For larger ships, this requires a large number of containers to be inspected, resulting in a heavy workload and low efficiency. Furthermore, the inspections require personnel to board the lashing bridge, posing a safety risk. Summary of the Invention

[0003] Embodiments of the present invention provide a container inspection method designed to address the low efficiency of existing manual inspections due to the large number of containers to be inspected and the heavy workload. This method constructs a sequence of feature matrices to be processed using the image eigenvalues ​​of the containers to be inspected. Spatial feature vectors are extracted from this sequence, and spatiotemporal feature vectors are extracted using these spatial feature vectors. This method enables automatic container inspection in both the spatiotemporal and temporal dimensions, improving inspection efficiency while also eliminating the need for manual inspection and enhancing safety.

[0004] In a first aspect, an embodiment of the present invention provides a container detection method, the method comprising:

[0005] Acquire a sequence of feature matrices to be processed, wherein the sequence of feature matrices to be processed includes feature matrices to be processed arranged according to a time dimension, and the feature matrices to be processed include image feature values ​​of a plurality of containers to be inspected arranged according to an adjacency relationship;

[0006] Performing a first feature extraction process on each frame of the feature matrix to be processed in the feature matrix sequence to be processed to obtain a spatial feature vector of the feature matrix sequence to be processed;

[0007] Performing a second feature extraction process on the spatial feature vector to obtain a spatiotemporal feature vector of the feature matrix sequence to be processed;

[0008] The detection results of the plurality of containers to be detected are determined according to the spatiotemporal feature vectors.

[0009] Optionally, obtaining the feature matrix to be processed includes:

[0010] When the number of adjacent containers to be inspected is greater than a preset number, acquiring a sequence of image matrices of the plurality of containers to be inspected within a preset time, the sequence of image matrices comprising image matrices arranged according to the time dimension, the image matrices comprising target images of the plurality of containers to be inspected arranged according to an adjacent relationship;

[0011] Performing image feature extraction processing on each target image in each frame image matrix to obtain image feature values ​​of each target image in each frame image matrix;

[0012] Based on the image feature values ​​of each target image in the image matrix of each frame, a feature matrix to be processed corresponding to the image matrix of each frame is determined.

[0013] Optionally, the step of acquiring a plurality of image matrix sequences of containers to be inspected within a preset time includes:

[0014] Within a preset time, a plurality of containers to be inspected are periodically photographed in a horizontal direction by means of sliding photographing equipment installed at different heights on the lashing bridge to obtain a sequence of original photographed images of each container to be inspected;

[0015] Determining a target image of each container to be inspected in each shooting cycle according to a sequence of original captured images of each container to be inspected;

[0016] The image matrix sequence is determined according to the target image of each container to be inspected in each shooting cycle and the adjacency relationship between the containers to be inspected.

[0017] Optionally, performing image feature extraction processing on each target image in each frame image matrix to obtain image feature values ​​of each target image in each frame image matrix includes:

[0018] Performing image feature extraction processing on each of the target images using a trained feature extraction model to obtain at least one of an offset feature, an angle feature, and a lashing feature of the container to be detected in each of the target images;

[0019] An image feature value of each target image in each frame image matrix is ​​determined according to at least one of the offset feature, the angle feature, and the binding feature corresponding to each target image.

[0020] Optionally, performing a first feature extraction process on each frame of the feature matrix to be processed in the feature matrix sequence to be processed to obtain a spatial feature vector of the feature matrix sequence to be processed includes:

[0021] Performing convolution processing on the feature matrix to be processed in each frame to obtain spatial features of the feature matrix to be processed in each frame;

[0022] Expanding the spatial features of the feature matrix to be processed in each frame in one dimension to obtain expanded spatial features;

[0023] The expanded spatial features are spliced ​​in the time dimension to obtain the spatial feature vectors of the feature matrix sequence to be processed.

[0024] Optionally, performing a second feature extraction process on the spatial feature vector to obtain the spatiotemporal feature vector of the feature matrix sequence to be processed includes:

[0025] Splitting the spatial feature vector according to a preset length to obtain a plurality of spatial feature sub-vectors;

[0026] The spatiotemporal feature vectors of the feature matrix sequence to be processed are obtained by performing a second feature extraction process on the plurality of spatial feature sub-vectors.

[0027] Optionally, performing a second feature extraction process on the plurality of spatial feature subvectors to obtain the spatiotemporal feature vectors of the feature matrix sequence to be processed includes:

[0028] Mapping each of the spatial feature subvectors into respective preset spaces through a preset mapping path to obtain a target vector space, wherein the target vector space includes a target feature subvector obtained by mapping the spatial feature subvectors, each mapping path is used to map one of the spatial feature subvectors into one of the target feature subvectors, and one target vector space corresponds to one matrix unit in the feature matrix to be processed;

[0029] Performing weighted fusion on all target feature subvectors in each of the target vector spaces to obtain fused feature vectors corresponding to each of the target vector spaces;

[0030] Performing a nonlinear transformation on the fused feature vector to obtain a spatiotemporal feature vector corresponding to the target vector space;

[0031] The spatiotemporal eigenvectors of the feature matrix sequence to be processed are determined according to the spatiotemporal eigenvectors corresponding to the target vector space.

[0032] In a second aspect, an embodiment of the present invention further provides a container detection device, the container detection device comprising:

[0033] An acquisition module is configured to acquire a sequence of feature matrices to be processed, wherein the sequence of feature matrices to be processed includes feature matrices to be processed arranged according to a time dimension, and the feature matrices to be processed include image feature values ​​of a plurality of containers to be inspected arranged according to an adjacency relationship;

[0034] A first processing module is configured to perform a first feature extraction process on each frame of the feature matrix to be processed in the feature matrix sequence to be processed, so as to obtain a spatial feature vector of the feature matrix sequence to be processed;

[0035] A second processing module is used to perform a second feature extraction process on the spatial feature vector to obtain the spatiotemporal feature vector of the feature matrix sequence to be processed;

[0036] A determination module is used to determine the detection results of the multiple containers to be detected according to the spatiotemporal feature vector.

[0037] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the container detection method provided in the embodiment of the present invention are implemented.

[0038] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the container detection method provided in the embodiment of the invention are implemented.

[0039] In an embodiment of the present invention, a sequence of feature matrices to be processed is obtained, wherein the sequence of feature matrices to be processed includes feature matrices to be processed arranged in a time dimension, and the feature matrices to be processed include image feature values ​​of multiple containers to be inspected arranged in an adjacency relationship; a first feature extraction process is performed on each frame of the feature matrix to be processed in the sequence of feature matrices to be processed to obtain a spatial feature vector of the sequence of feature matrices to be processed; a second feature extraction process is performed on the spatial feature vector to obtain a spatiotemporal feature vector of the sequence of feature matrices to be processed; and the inspection results of the multiple containers to be inspected are determined based on the spatiotemporal feature vector. The sequence of feature matrices to be processed is constructed using the image feature values ​​of the containers to be inspected, and the spatial feature vector is extracted from the sequence of feature matrices to be processed. The spatiotemporal feature vector is extracted using the spatial feature vector, so that the containers can be automatically inspected in the spatiotemporal dimension, thereby improving the efficiency of container inspection. At the same time, manual inspection is not required, thereby improving the safety of container inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1This is a flow chart of a container detection method provided by an embodiment of the present invention;

[0042] Figure 2 1 is a schematic structural diagram of a container detection device provided in an embodiment of the present invention;

[0043] Figure 3 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0045] like Figure 1 As shown, Figure 1 This is a flow chart of a container detection method provided by an embodiment of the present invention. The container detection method includes the following steps:

[0046] 101. Obtain a feature matrix sequence to be processed.

[0047] In an embodiment of the present invention, the above-mentioned sequence of feature matrices to be processed includes a feature matrix to be processed arranged according to the time dimension, and the above-mentioned feature matrix to be processed includes image feature values ​​of multiple containers to be inspected arranged according to an adjacency relationship. The above-mentioned sequence of feature matrices to be processed can be represented by M×N×R, where the above-mentioned feature matrix to be processed is M×N and the above-mentioned time dimension is R. The feature matrix to be processed can be understood as a matrix with M rows and N columns, and there are M×N matrix units in the feature matrix to be processed. Each container to be inspected corresponds to a matrix unit, and the value of the matrix unit is the image feature value of the corresponding container to be inspected. If a matrix unit does not correspond to a container to be inspected, the value of the matrix unit is 0 or null. The above-mentioned sequence of feature matrices to be processed can be understood as a three-dimensional matrix. The above-mentioned multiple containers to be inspected are multiple containers to be inspected that are tied together.

[0048] The aforementioned adjacency relationship can be understood as an adjacent relationship between two containers. This adjacent relationship can include left, right, top, and bottom adjacent relationships. Since containers have a hexahedral structure, they can be stacked in rows horizontally and columns vertically on a ship. Two adjacent containers in the same row have a left-right adjacent relationship, and two adjacent containers in the same column have a top-bottom adjacent relationship. In an embodiment of the present invention, on the side facing the camera, a maximum of N containers are placed in each row, and a maximum of M rows are placed. That is, the stacking structure of the containers corresponds to the feature matrix to be processed. Thus, image feature values ​​can be added to the corresponding matrix cells based on the position of the containers.

[0049] After multiple containers to be inspected are stacked and tied, they can be photographed using a camera to obtain images of the containers. Target detection can be performed on the images to obtain a detection frame (x, y, w, h, r) for each container in the image. (x, y) represents the center of the detection frame, w represents the width of the detection frame, h represents the height of the detection frame, and r represents the confidence level that the detection frame is a container. The image within the detection frame corresponds to the container to be inspected. Image feature extraction can be performed on the image of the container to be inspected to obtain image feature values ​​of the container to be inspected. Alternatively, the detection frame can be expanded to obtain an expanded image of the container to be inspected. Image feature extraction can then be performed on the expanded image of the container to obtain image feature values ​​of the container to be inspected.

[0050] The above-mentioned image feature extraction may be extracting at least one of the offset feature, angle feature and lashing feature of the container to be detected, and the above-mentioned image feature value may be an image feature value determined based on at least one of the offset feature, angle feature and lashing feature of the container to be detected.

[0051] Multiple containers to be inspected can be photographed by a photographing device at preset intervals to obtain images taken at different times. Target detection and image feature extraction are performed on the images taken at different times, and the image feature values ​​of each container to be inspected at different times are added to the feature matrix to be processed of the corresponding time dimension to obtain a sequence of feature matrices to be processed.

[0052] 102. Perform a first feature extraction process on each frame of the feature matrix to be processed in the feature matrix sequence to be processed to obtain a spatial feature vector of the feature matrix sequence to be processed.

[0053] In an embodiment of the present invention, the sequence of feature matrices to be processed includes R M×N feature matrices to be processed, arranged along the time dimension. Each matrix cell in the matrix has a value, which may be the image feature value of the corresponding container to be inspected. If a matrix cell does not correspond to a container to be inspected, the value of the matrix cell is 0 or null. The sequence of feature matrices to be processed can be understood as a three-dimensional matrix.

[0054] It is understandable that, since the stacking of containers is continuous, adjacent containers may have an implicit relationship of mutual influence. Therefore, the implicit relationship of mutual influence can be represented by the spatial eigenvector of the feature matrix to be processed.

[0055] The feature matrix to be processed can be regarded as an image with a size of M×N, and the first feature extraction process can be a feature extraction process implemented by feature extraction methods such as Harris, SURF, LBF, HOG, DPM, and ORB.

[0056] Specifically, by performing the first feature extraction process on the i-th feature matrix to be processed through the feature extraction method, the space vector a of the i-th feature matrix to be processed can be obtained. i , perform the first feature extraction process on all feature matrices to be processed, and then the spatial feature vector a of the feature matrix sequence to be processed can be obtained R =(a1,…,a i ,…,a R ).

[0057] 103. Perform a second feature extraction process on the spatial feature vector to obtain a spatiotemporal feature vector of the feature matrix sequence to be processed.

[0058] In an embodiment of the present invention, after obtaining the spatial feature vector of the feature matrix sequence to be processed, a timing network can be used to extract the temporal dependency in the spatial feature vector to obtain the spatiotemporal feature vector of the feature matrix sequence to be processed. The above-mentioned timing network can be a timing network based on a recurrent neural network RNN, a long short-term memory network LSTM, or the like.

[0059] Through the above temporal network, the spatial feature vector a R =(a1,…,a i ,…,a R ) performs the second feature extraction process to obtain the spatiotemporal feature vector b of the feature matrix sequence to be processed M×N =(b 1,1 , b 1,2 …, b i,j ,…,b M×(N-1) , b M×N), the above spatiotemporal feature vector is fused with the spatial feature vector a R =(a1,…,a i ,…,a R ) in the timing dependency, b i,j Represents the space-time vector corresponding to the matrix unit in the i-th row and j-th column in the feature matrix to be processed.

[0060] It can be understood that the output of the above-mentioned timing network is constructed as M×N time-space vectors, each time-space vector corresponds to a matrix unit of a feature matrix to be processed, that is, whether the corresponding container to be detected is abnormal can be determined by the time-space vector of the matrix unit.

[0061] 104. Determine detection results of multiple containers to be detected based on the spatiotemporal feature vectors.

[0062] In the embodiment of the present invention, after obtaining the spatiotemporal eigenvector b of the feature matrix sequence to be processed M×N =(b 1,1 , b 1,2 …, b i,j ,…,b M×(N-1) , b M×N ) can be obtained after the time-space vector b of each matrix unit in the feature matrix to be processed i,j , in determining the space-time vector b of each matrix unit i,j After that, the space-time vector b can be classified by the classification model i,j Classify and get the space-time vector b i,j The classification results can be of two types: normal detection and abnormal detection.

[0063] The above classification model can be a classification model based on support vector machine SVM, logistic regression LR, decision tree CART, random forest RF, artificial neural network ANN, etc.

[0064] The classification results can be returned to the feature matrix to be processed to obtain a result matrix, which includes the classification results of each container to be inspected. Furthermore, the result matrix can differentiate between the two types of results: normal detection and abnormal detection.

[0065] In an embodiment of the present invention, a sequence of feature matrices to be processed is obtained, the sequence of feature matrices to be processed comprising feature matrices to be processed arranged in a time dimension, and the feature matrices to be processed comprising image feature values ​​of a plurality of containers to be inspected arranged in an adjacency relationship; a first feature extraction process is performed on each frame of the feature matrix to be processed in the sequence of feature matrices to be processed to obtain a spatial feature vector of the sequence of feature matrices to be processed; a second feature extraction process is performed on the spatial feature vector to obtain a spatiotemporal feature vector of the sequence of feature matrices to be processed; and detection results of the plurality of containers to be inspected are determined based on the spatiotemporal feature vector. The sequence of feature matrices to be processed is constructed using the image feature values ​​of the containers to be inspected, the spatial feature vector is extracted from the sequence of feature matrices to be processed, and the spatiotemporal feature vector is extracted using the spatial feature vector. The containers can be automatically inspected in the spatiotemporal dimension, thereby improving the efficiency of container inspection and, at the same time, eliminating the need for manual inspection, thereby improving the safety of container inspection.

[0066] It is understandable that in the specific implementation of this application, related data such as images of ships or ports are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0067] Optionally, in the step of obtaining a sequence of feature matrices to be processed, when the number of containers to be inspected with an adjacency relationship is greater than a preset number, an image matrix sequence of multiple containers to be inspected can be obtained within a preset time, the image matrix sequence including an image matrix arranged according to a time dimension, and the image matrix including multiple target images of containers to be inspected arranged according to an adjacency relationship; image feature extraction processing is performed on each target image in each frame of the image matrix to obtain an image feature value of each target image in each frame of the image matrix; based on the image feature value of each target image in each frame of the image matrix, the feature matrix to be processed corresponding to each frame of the image matrix is ​​determined; based on the feature matrix to be processed corresponding to each frame of the image matrix, a sequence of feature matrices to be processed is determined.

[0068] In this embodiment of the present invention, the containers to be inspected are lashed containers. The aforementioned adjacency relationship can be understood as an adjacency relationship between two containers. This adjacency relationship can include left, right, top, and bottom adjacencies. Since containers are hexahedral structures, they can be stacked in horizontal rows and vertical columns on a ship. Adjacent containers in a row have a left-right adjacency relationship, and adjacent containers in a column have a top-bottom adjacency relationship. The preset number is less than or equal to M×N. For example, the preset number can be 80% of M×N.

[0069] When the number of tied containers exceeds a preset number, a camera can be used to photograph the tied containers, obtaining images of multiple containers to be inspected. An initial matrix of size M×N can be determined as a matrix template. After obtaining the image eigenvalues, the image eigenvalues ​​are added to the matrix cells at the corresponding positions in the matrix template to obtain the feature matrix to be processed.

[0070] Specifically, a single camera can be used to capture multiple images of the tied containers within the predicted timeframe. Each capture produces an image containing multiple containers to be inspected. Each captured image includes multiple containers to be inspected. Therefore, a captured image can be considered a frame matrix. The captured images corresponding to the multiple captures can be arranged according to the time dimension to produce an image matrix sequence, where each frame matrix in the image matrix sequence is a captured image. The time dimension is determined based on the number of captures and the capture time. For example, the captured images corresponding to the multiple captures are arranged according to the capture time, with the number of captures representing the length of the time dimension. If the number of captures is R, then the length of the time dimension is R, and the capture time represents the granularity of the time dimension. Each captured image is preprocessed to obtain a preprocessed captured image. This preprocessing can include spatial transformation, denoising, and other preprocessing methods. Target detection is performed on the preprocessed captured images to obtain a detection box (x, y, w, h, r) for each container to be inspected during each capture. The image within the detection box is the target image of the container to be inspected. Image feature extraction can be performed on the target image of the container to be inspected to obtain image feature values ​​for the container to be inspected. Alternatively, the inspection frame can be expanded to obtain an expanded target image of the container to be inspected. Image features are then extracted from this expanded target image to obtain image feature values ​​for the container to be inspected. The image feature values ​​of each container to be inspected in each shot are added to the corresponding matrix cells in the initial matrix to obtain a feature matrix to be processed for each shot. After obtaining the feature matrix to be processed for each shot, the feature matrices are arranged along the time dimension to obtain a sequence of feature matrices to be processed.

[0071] Alternatively, multiple cameras can be used to photograph the tied containers. For example, one camera can be used to photograph each container, resulting in target images corresponding to each container to be inspected. Each target image includes an image of the container to be inspected and images of adjacent portions of adjacent containers. Specifically, multiple cameras can be used to photograph the tied containers multiple times within the predicted timeframe. Each shot produces multiple target images corresponding to the containers to be inspected. All target images obtained from each shot are then spliced ​​together based on the adjacency relationship between the containers to be inspected, resulting in an image matrix corresponding to each shot. Each shot corresponds to a frame of the image matrix. The image matrices corresponding to the multiple shots can be arranged according to a time dimension, resulting in an image matrix sequence. The time dimension is determined by the number of shots and the shooting time. For example, the image matrices corresponding to the multiple shots can be arranged according to the shooting time, with the number of shots representing the length of the time dimension. If the number of shots is R, then the length of the time dimension is R, and the shooting time represents the granularity of the time dimension. Image feature extraction can be performed on the target images in the image matrix to obtain image feature values ​​for the containers to be inspected. The image feature values ​​of each container to be detected in each frame of the image matrix are added to the corresponding matrix unit in the initial matrix to obtain the feature matrix to be processed corresponding to each shooting. After processing all the image matrices in the image matrix sequence, a sequence of feature matrices to be processed is obtained.

[0072] In one possible embodiment, after obtaining target images corresponding to multiple containers to be inspected during each capture, image features can be directly extracted from each captured image to obtain image features for each container to be inspected. The image features of each container to be inspected during each capture are then added to the corresponding matrix cells in the initial matrix to obtain a feature matrix to be processed corresponding to each capture. After obtaining the feature matrix to be processed corresponding to each capture, the feature matrices to be processed are arranged according to the time dimension to obtain a sequence of feature matrices to be processed.

[0073] The preset time begins when the number of adjacent containers to be inspected exceeds a preset number. The preset time can be determined based on the time between the completion of container loading and the completion of ship departure preparations. The preset time is less than the time between the completion of container loading and the completion of ship departure preparations. For example, if the time between the completion of container loading and the completion of ship departure preparations is 48 hours, the preset time can be 40 hours. Within the preset time, the lashed containers can be photographed R times. This allows for continuous container inspections before the ship departs, resulting in more reliable inspection results.

[0074] In an embodiment of the present invention, on the side facing the shooting device, a maximum of N containers are placed in each row, and a maximum of M rows are placed, that is, the stacking structure of the containers corresponds to the feature matrix to be processed. In this way, image feature values ​​can be added to the corresponding matrix units according to the positions of the containers.

[0075] Optionally, in the step of obtaining an image matrix sequence of multiple containers to be inspected within a preset time, the multiple containers to be inspected can be periodically slid and photographed in the horizontal direction within the preset time by means of sliding photographing equipment installed at different heights on the lashing bridge to obtain an original photographed image sequence of each container to be inspected; based on the original photographed image sequence of each container to be inspected, a target image of each container to be inspected in each photographing cycle is determined; and based on the target image of each container to be inspected in each photographing cycle and the adjacency relationship between each container to be inspected, an image matrix sequence is determined.

[0076] In this embodiment of the present invention, containers are secured by lashing bridges. Horizontal slide rails parallel to the container plane are installed at different heights on the lashing bridges. A sliding camera is mounted on the slide rails and slides along them, enabling periodic horizontal filming of a row of containers to be inspected. Because sliding filming is a continuous process, the predicted time can be divided into R filming cycles.

[0077] It should be noted that using a single camera to capture multiple containers to be inspected in the same image requires the camera to have a high resolution to ensure the accuracy of target detection and image feature extraction. A small increase in resolution will significantly increase the cost of investment. The accuracy of existing algorithms for target detection and image feature extraction in low-resolution images also faces bottlenecks. Only when there are no significant breakthroughs in hardware costs and software bottlenecks can the use of multiple cameras for target detection and image feature extraction in the same image be popularized. Using multiple cameras to capture multiple containers to be inspected separately is equivalent to increasing the number of cameras to obtain target images with higher resolution, which is costly. However, the embodiments of the present invention use sliding cameras and rails to capture target images of each container to be inspected with fewer devices while ensuring the shooting resolution.

[0078] During the sliding shooting process, the sliding shooting device shoots at a preset frame rate. After one shooting cycle is completed, a shooting image sequence with the same length as the shooting cycle will be obtained. The shooting image sequence can be divided according to the container to be inspected to obtain the original shooting image sequence of each container to be inspected. In the original shooting image sequence of each container to be inspected, the image with the highest image quality is extracted as the target image of the container to be inspected in the shooting cycle. The above-mentioned image quality can be obtained by performing image quality evaluation on each frame image in the original shooting image sequence through an image quality evaluation model. The above-mentioned image quality evaluation model is used to evaluate the image quality of the image of the container to be inspected and the adjacent part image of the adjacent container in the frame image, and output a quality score. The above-mentioned image quality evaluation model can be a model based on a deep convolutional neural network, for example, it can be a model based on a convolutional neural network such as an R-CNN network, a Faster R-CNN network, a YOLO network, or the like. The above-mentioned image quality assessment model can be obtained by supervised training of a large number of sample container images labeled with quality scores. Specifically, a quality assessment model to be trained is constructed, sample container images are collected, and the quality scores of the sample container images are labeled to obtain quality score labels for the sample container images. The sample container images are input into the quality assessment model to be trained to obtain quality score results for the sample container images. The error loss between the quality score results of the sample container images and the quality score labels of the sample container images is calculated. With minimizing the error loss as the optimization goal, the parameters of the quality assessment model to be trained are adjusted through backpropagation. The above-mentioned training process is iterated until the quality assessment model to be trained converges at the minimum error loss or the number of iterations reaches a preset number. Training is then stopped to obtain a trained quality assessment model. Each frame of the original image sequence of the container to be inspected is input into the quality assessment model, and the quality score of each frame is output. The frame with the highest quality score is determined as the target image of the container to be inspected.

[0079] The target images of all containers to be inspected during the capture cycle are spliced ​​together based on their adjacency to produce an image matrix. The image matrices from different capture cycles are then arranged in chronological order to produce an image matrix sequence. Once the image matrix is ​​obtained, image features can be extracted from the target images in the image matrix to obtain the image eigenvalues ​​of the containers to be inspected. The image eigenvalues ​​of each container to be inspected in each frame of the image matrix are added to the corresponding matrix cells in the initial matrix to produce the feature matrix to be processed for each capture.

[0080] The fixed work is completed by professionals engaged in lashing work, and the number of containers to be detected with adjacent relationships can be automatically detected. When the number of containers to be detected with adjacent relationships is greater than the preset number.

[0081] Optionally, in the step of performing image feature extraction processing on each target image in each frame image matrix to obtain the image feature value of each target image in each frame image matrix, image feature extraction processing can be performed on each target image through a trained feature extraction model to obtain at least one of the offset feature, angle feature and lashing feature of the container to be detected in each target image; and the image feature value of each target image in each frame image matrix is ​​determined based on at least one of the offset feature, angle feature and lashing feature corresponding to each target image.

[0082] In an embodiment of the present invention, the feature extraction model may be a feature extraction model based on feature extraction methods such as Harris, SURF, LBF, HOG, DPM, and ORB.

[0083] The above-mentioned feature extraction model can also be a feature extraction model based on a deep convolutional neural network. Specifically, a training data set and a feature extraction model to be trained can be obtained. The training data set includes sample images. The method of obtaining the sample images is the same as the method of obtaining the above-mentioned image matrix. Each sample image corresponds to a set of labels, and the set of labels includes at least one of an offset label, an angle label, and a lashing label. Among them, the offset label is obtained by marking the offset of each container in the sample image; the angle label is obtained by marking the angle between the center line of each container in the sample image and the horizontal plane, and the horizontal plane is the relative horizontal plane of the shooting device; the lashing label is obtained by marking the lashing conditions of each container in the sample image. The specific labels in a set of labels can be selected according to the actual situation. When only one feature needs to be extracted as the image feature value, the set of labels can only include the label corresponding to the feature type, and the feature extraction model to be trained can only construct the output structure of the feature type. For example, when only offset features are needed as image feature values, a set of labels can include only offset labels, and the feature extraction model to be trained can only construct the output structure of offset features. When only angle features need to be extracted as image feature values, a set of labels can include only angle labels, and the feature extraction model to be trained can only construct the output structure of angle features. When only binding features need to be extracted as image feature values, a set of labels can include only binding labels, and the feature extraction model to be trained can only construct the output structure of binding features. When two features need to be extracted to determine image feature values, a set of labels can include labels corresponding to the two feature types, and the feature extraction model to be trained can only construct the output structures of the two feature types. For example, when it is necessary to extract offset features and angle features to determine image feature values, a set of labels can include offset labels and angle labels, and the feature extraction model to be trained can only construct the output structure of offset features and angle features; when it is necessary to extract offset features and binding features to determine image feature values, a set of labels can include offset labels and binding labels, and the feature extraction model to be trained can only construct the output structure of offset features and binding features; when it is necessary to extract angle features and binding features to determine image feature values, a set of labels can include angle labels and binding labels, and the feature extraction model to be trained can only construct the output structure of angle features and binding features. When it is necessary to extract offset features, angle features, and binding features to determine image feature values, a set of labels can include offset labels, angle labels, and binding labels, and the feature extraction model to be trained can construct the output structure of offset features, angle features, and binding features.During the training process, a sample image is input into the feature extraction model to be trained, and a feature result of the sample image is obtained. The error loss between the feature result of the sample image and the label of the sample image is calculated. With minimizing the error loss as the optimization goal, the parameters of the feature extraction model to be trained are adjusted through backpropagation. The above training process is iterated until the feature extraction model to be trained converges at the minimum error loss or the number of iterations reaches a preset number. Training is then stopped to obtain a trained feature extraction model. Each frame image matrix is ​​input into the trained feature extraction model, and at least one of the offset features, angle features, and binding features corresponding to each target image in each frame image matrix is ​​output.

[0084] In one possible embodiment, the trained feature extraction model outputs a single type of feature corresponding to each target image, and the feature of that type for each target image can be used as the image feature value of each target image. For example, if the trained feature extraction model outputs an offset feature corresponding to each target image, the offset feature corresponding to each target image can be used as the image feature value of each target image. If the trained feature extraction model outputs an angle feature corresponding to each target image, the angle feature corresponding to each target image can be used as the image feature value of each target image. If the trained feature extraction model outputs a binding feature corresponding to each target image, the binding feature corresponding to each target image can be used as the image feature value of each target image.

[0085] In a possible embodiment, the trained feature extraction model outputs two types of features corresponding to each target image, and the image feature value of each target image can be determined based on the two types of features of each target image. For example, if the trained feature extraction model outputs offset features and angle features corresponding to each target image, the image feature value of each target image can be determined based on the offset features and angle features corresponding to each target image. If the trained feature extraction model outputs angle features and binding features corresponding to each target image, the image feature value of each target image can be determined based on the angle features and binding features corresponding to each target image. If the trained feature extraction model outputs offset features and binding features corresponding to each target image, the image feature value of each target image can be determined based on the offset features and binding features corresponding to each target image. Specifically, the result of adding the two types of features can be used as the image feature value of the corresponding target image, or the two types of features can be spliced ​​to obtain the spliced ​​result as the image feature value of the corresponding target image.

[0086] In one possible embodiment, the trained feature extraction model outputs the offset features, angle features, and binding features corresponding to each target image. The sum of the offset features, angle features, and binding features can be used as the image feature value of the corresponding target image, or the offset features, angle features, and binding features can be spliced ​​to obtain the spliced ​​result as the image feature value of the corresponding target image.

[0087] Optionally, in the step of performing first feature extraction processing on each frame of the feature matrix to be processed in the feature matrix sequence to be processed to obtain the spatial feature vector of the feature matrix sequence to be processed, convolution processing can be performed on each frame of the feature matrix to be processed to obtain the spatial features of each frame of the feature matrix to be processed; the spatial features of each frame of the feature matrix to be processed can be expanded in one dimension to obtain the expanded spatial features; and the expanded spatial features can be spliced ​​in the time dimension to obtain the spatial feature vector of the feature matrix sequence to be processed.

[0088] In an embodiment of the present invention, a preset convolution kernel can be used to perform convolution processing on the M×N feature matrix to be processed for each frame. The size of the convolution kernel can be a 3×3 convolution kernel. By performing convolution processing on the M×N feature matrix to be processed through the convolution kernel, the spatial relationship between each matrix unit and the surrounding matrix units can be extracted. For example, a 3×3 convolution kernel can extract the spatial relationship between the central matrix unit and the surrounding 8 matrix units, thereby obtaining the spatial features of the feature matrix to be processed. The parameters of the convolution kernel can be fixed parameters set artificially, or the parameters of the convolution kernel can be parameters obtained through training. The training of the convolution kernel is joint training. Specifically, the parameters of the convolution kernel can be obtained by joint training with the detection results.

[0089] After obtaining the spatial features of each frame of the feature matrix to be processed, the spatial features of each frame of the feature matrix to be processed can be expanded in one dimension. The expanded spatial features are represented by a one-dimensional vector. The expanded spatial features can also be called spatial vectors. The expanded spatial features are sequentially spliced ​​in the time dimension to obtain the spatial feature vector a of the feature matrix sequence to be processed. R =(a1,…,a i ,…,a R ).

[0090] Optionally, in the step of performing a second feature extraction process on the spatial feature vector to obtain the spatiotemporal feature vector of the feature matrix sequence to be processed, the spatial feature vector can be divided according to a preset length to obtain multiple spatial feature sub-vectors; and by performing a second feature extraction process on multiple spatial feature sub-vectors, the spatiotemporal feature vector of the feature matrix sequence to be processed is obtained.

[0091] In the embodiment of the present invention, after obtaining the spatial feature vector aR =(a1,…,a i ,…,a R ) after which the spatial eigenvector a R =(a1,…,a i ,…,a R ) to obtain K spatial feature sub-vectors, K is less than R, and each spatial feature sub-vector can be obtained by c k =(c1, c2, ..., c t ), k represents the kth spatial feature subvector, each spatial feature subvector may include t consecutive spatial vectors, and t is greater than or equal to 2. It can be understood that R=K×t.

[0092] After obtaining K spatial feature sub-vectors, since the spatial feature sub-vectors contain the temporal information of adjacent spatial vectors, a temporal network can be used to extract the temporal dependencies in each spatial feature sub-vector to obtain the spatiotemporal feature vectors of the feature matrix sequence to be processed. The above-mentioned temporal network can be based on a temporal network such as a recurrent neural network RNN ​​or a long short-term memory network LSTM.

[0093] Specifically, the spatial eigenvector c k =(c1, c2, ..., c t ) are input into the time series network in sequence to obtain the spatiotemporal feature vector b of the feature matrix sequence to be processed M×N =(b 1,1 , b 1,2 …, b i,j ,…,b M×(N-1) , b M×N ), the above spatiotemporal feature vector is fused with the spatial feature sub-vector c k =(c1, c2, ..., c t ) and the temporal dependence between each spatial feature vector, so that the spatiotemporal feature vector b M×N =(b 1,1 , b 1,2 …, b i,j ,…,b M×(N-1) , b M×N ) is more accurate, b i,j Represents the space-time vector corresponding to the matrix unit in the i-th row and j-th column in the feature matrix to be processed.

[0094] Optionally, in the step of performing a second feature extraction process through multiple spatial feature subvectors to obtain the spatiotemporal feature vectors of the feature matrix sequence to be processed, each spatial feature subvector can be mapped to each preset space through a preset mapping path to obtain a target vector space, the target vector space including the target feature subvectors obtained by mapping the spatial feature subvectors, each mapping path is used to map a spatial feature subvector into a target feature subvector, and a target vector space corresponds to a matrix unit in the feature matrix to be processed; all target feature subvectors in each target vector space are weightedly fused to obtain a fused feature vector corresponding to each target vector space; the fused feature vector is nonlinearly transformed to obtain the spatiotemporal feature vector corresponding to the target vector space; and the spatiotemporal feature vector of the feature matrix sequence to be processed is determined based on the spatiotemporal feature vector corresponding to the target vector space.

[0095] In an embodiment of the present invention, the preset space is determined based on the matrix units in the feature matrix to be processed. The number of matrix units in the feature matrix to be processed is M×N, and the number of preset spaces is also M×N. Each preset space corresponds to a matrix unit, and each preset space corresponds to a container to be inspected.

[0096] The above mapping path includes mapping relationship and mapping weight. The mapping relationship is the mapping relationship between a spatial feature subvector and a preset space, and the mapping weight is the mapping weight between a spatial feature subvector and a preset space. For example, the mapping relationship is (i, j), which means that the i-th spatial feature subvector is mapped to the j-th preset space, and the mapping weight W is i,j Represents the mapping weight between the i-th spatial feature subvector and the j-th preset space. The target feature subvector is the weighted product of the spatial feature subvector and the mapping weight. Specifically, the target feature subvector d i,j =c i W i,j .

[0097] After mapping all spatial feature subvectors to the preset space, M×N target vector spaces are obtained, each of which includes K target feature subvectors. For any target vector space j, all target feature subvectors d i,j Perform weighted summation to obtain the fusion feature vector s corresponding to the target vector space j j .

[0098] Specifically, the fusion feature vector s j As shown in the following formula:

[0099]

[0100] The above e ijis the fusion weight, the above d j│i is the target feature subvector in the target vector space j.

[0101] According to the fusion feature vector s j After that, the fusion feature vector s j Perform nonlinear transformation to obtain the space-time feature vector v corresponding to the target vector space j The above nonlinear transformation can be expressed as follows:

[0102]

[0103] The above fusion weight e ij According to the spatiotemporal feature vector v j To update, the update process is as follows:

[0104]

[0105] f ij =f ij +d j│i ·v j

[0106] Among them, f ij is the intermediate parameter.

[0107] After the above nonlinear transformation, M×N spatiotemporal eigenvectors can be obtained, each of which corresponds to a matrix unit in the feature matrix to be processed. Specifically, after obtaining the spatiotemporal eigenvector v of the feature matrix sequence to be processed j After that, the spatiotemporal feature vector v can be calculated j The modulus of the space-time eigenvector v j If the modulus of the container to be detected is greater than or equal to the preset modulus, it can be determined that the detection result of the container to be detected corresponding to the j-th matrix unit is normal. j If the modulus of is less than the preset modulus, it can be determined that the detection result of the container to be detected corresponding to the j-th matrix unit is abnormal.

[0108] It should be noted that the above mapping weight W i,j According to the back propagation algorithm, the above fusion weights e ij According to the intermediate parameter f ij Update the intermediate parameter f ij According to the space-time feature vector v of the previous round j and the corresponding target feature vector d j│i Perform iterative updates. The parameters of the above convolution kernels can also be updated according to the back-propagation algorithm.

[0109] Specifically, during the training process, the input is a sample feature matrix, which is obtained in the same way as the feature matrix to be processed. Each matrix unit in the sample feature matrix corresponds to a sample label in the form of a vector. The sample feature matrix is ​​processed in sequence through the convolution kernel to be trained, the mapping weight to be trained, and the fusion weight to be trained to obtain the spatiotemporal feature vector of the sample feature matrix. The loss function between the spatiotemporal feature vector of the sample feature matrix and the sample label of the sample feature matrix is ​​calculated, and back propagation is performed through the loss function. The above-mentioned mapping weight W i,j The parameters of the convolution kernel are updated by back propagation algorithm with minimizing the loss function as the optimization goal, where the loss function is as follows:

[0110] L k =T k max(0, m + -||v k ||) 2 +λ·(1-T k )·max(0,||v k ||-m - ) 2

[0111] Among them, T k is the sample label of the kth sample container, m + is the upper boundary hyperparameter, m - is the lower boundary hyperparameter, λ is the hyperparameter, v k is the kth spatiotemporal eigenvector, max(0,m + -||v k ||) means 0 or m + -||v k ||, the larger one, max(0, ||v k ||-m - ) means 0 or ||v k ||-m - The larger one.

[0112] It should be noted that the container detection method provided in the embodiment of the present invention can be applied to devices such as camera equipment, smart phones, computers, servers, etc. that can perform container detection.

[0113] like Figure 2 As shown, an embodiment of the present invention provides a container detection device, which includes:

[0114] An acquisition module 201 is configured to acquire a sequence of feature matrices to be processed, wherein the sequence of feature matrices to be processed includes feature matrices to be processed arranged in a time dimension, and the feature matrices to be processed include image feature values ​​of a plurality of containers to be inspected arranged in an adjacency relationship;

[0115] A first processing module 202 is configured to perform a first feature extraction process on each frame of the feature matrix sequence to be processed to obtain a spatial feature vector of the feature matrix sequence to be processed;

[0116] The second processing module 203 is used to perform a second feature extraction process on the spatial feature vector to obtain the spatiotemporal feature vector of the feature matrix sequence to be processed;

[0117] The determination module 204 is configured to determine detection results of the plurality of containers to be detected based on the spatiotemporal feature vectors.

[0118] Optionally, the acquisition module 201 includes:

[0119] an acquisition submodule, configured to acquire, within a preset time period, an image matrix sequence of a plurality of containers to be inspected that are adjacent to each other, when the number of containers to be inspected that are adjacent to each other is greater than a preset number, the image matrix sequence comprising an image matrix arranged according to the time dimension, the image matrix comprising target images of the plurality of containers to be inspected that are arranged according to an adjacent relationship;

[0120] The first processing submodule is used to perform image feature extraction processing on each target image in each frame image matrix to obtain image feature values ​​of each target image in each frame image matrix;

[0121] A first determining submodule is configured to determine a feature matrix to be processed corresponding to each frame of the image matrix based on the image feature values ​​of each target image in each frame of the image matrix;

[0122] The second determining submodule is configured to determine a sequence of feature matrices to be processed based on the feature matrix to be processed corresponding to each frame of the image matrix.

[0123] Optionally, the acquisition submodule includes:

[0124] a photographing unit, configured to periodically slide photograph the plurality of containers to be inspected in a horizontal direction within a preset time using a sliding photographing device installed at different heights on the lashing bridge, to obtain a sequence of original photographed images of each container to be inspected;

[0125] A first determining unit is configured to determine a target image of each container to be inspected in each shooting cycle according to a sequence of original captured images of each container to be inspected;

[0126] The second determining unit is configured to determine the image matrix sequence according to the target image of each container to be inspected in each shooting cycle and the adjacency relationship between the containers to be inspected.

[0127] Optionally, the first processing submodule includes:

[0128] a first processing unit, configured to perform image feature extraction processing on each of the target images using a trained feature extraction model to obtain at least one of an offset feature, an angle feature, and a lashing feature of the container to be detected in each of the target images;

[0129] The third determining unit is configured to determine an image feature value of each target image in each frame image matrix according to at least one of the offset feature, the angle feature, and the binding feature corresponding to each target image.

[0130] Optionally, the first processing module 202 includes:

[0131] The second processing submodule is configured to perform convolution processing on the feature matrix to be processed in each frame to obtain spatial features of the feature matrix to be processed in each frame;

[0132] A third processing submodule is configured to perform one-dimensional expansion on the spatial features of the feature matrix to be processed in each frame to obtain expanded spatial features;

[0133] The fourth processing submodule is configured to splice the expanded spatial features in a time dimension to obtain spatial feature vectors of the feature matrix sequence to be processed.

[0134] Optionally, the second processing module 203 includes:

[0135] A sub-slicing module, configured to split the spatial feature vector into sub-vectors according to a preset length to obtain a plurality of spatial feature sub-vectors;

[0136] The fifth processing submodule is configured to obtain the spatiotemporal feature vectors of the feature matrix sequence to be processed by performing a second feature extraction process on the plurality of spatial feature subvectors.

[0137] Optionally, the fifth processing submodule includes:

[0138] a second processing unit, configured to map each of the spatial feature subvectors into respective preset spaces through a preset mapping path to obtain a target vector space, wherein the target vector space includes a target feature subvector obtained by mapping the spatial feature subvectors, each mapping path being configured to map one of the spatial feature subvectors into one of the target feature subvectors, and one target vector space corresponding to one matrix unit in the feature matrix to be processed;

[0139] A fusion unit, configured to perform weighted fusion on all target feature subvectors in each of the target vector spaces to obtain a fused feature vector corresponding to each of the target vector spaces;

[0140] a transformation unit, configured to perform a nonlinear transformation on the fused feature vector to obtain a spatiotemporal feature vector corresponding to the target vector space;

[0141] The fourth determining unit is configured to determine the spatiotemporal eigenvector of the feature matrix sequence to be processed according to the spatiotemporal eigenvector corresponding to the target vector space.

[0142] It should be noted that the container detection device provided in the embodiment of the present invention can be applied to devices such as camera equipment, smart phones, computers, servers, etc. that can perform container detection.

[0143] The container detection device provided in the embodiment of the present invention can implement each process implemented by the container detection method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0144] See also Figure 3 , Figure 3 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, it includes: a memory 302, a processor 301, and a computer program of a container detection method stored in the memory 302 and executable on the processor 301, wherein:

[0145] The processor 301 is configured to call the computer program stored in the memory 302 and execute the following steps:

[0146] Acquire a sequence of feature matrices to be processed, wherein the sequence of feature matrices to be processed includes feature matrices to be processed arranged according to a time dimension, and the feature matrices to be processed include image feature values ​​of a plurality of containers to be inspected arranged according to an adjacency relationship;

[0147] Performing a first feature extraction process on each frame of the feature matrix to be processed in the feature matrix sequence to be processed to obtain a spatial feature vector of the feature matrix sequence to be processed;

[0148] Performing a second feature extraction process on the spatial feature vector to obtain a spatiotemporal feature vector of the feature matrix sequence to be processed;

[0149] The detection results of the plurality of containers to be detected are determined according to the spatiotemporal feature vectors.

[0150] Optionally, the obtaining of the feature matrix to be processed performed by the processor 301 includes:

[0151] When the number of adjacent containers to be inspected is greater than a preset number, acquiring a sequence of image matrices of the plurality of containers to be inspected within a preset time, the sequence of image matrices comprising image matrices arranged according to the time dimension, the image matrices comprising target images of the plurality of containers to be inspected arranged according to an adjacent relationship;

[0152] Performing image feature extraction processing on each target image in each frame image matrix to obtain image feature values ​​of each target image in each frame image matrix;

[0153] Determine a feature matrix to be processed corresponding to each frame of the image matrix based on the image feature values ​​of each target image in each frame of the image matrix;

[0154] Based on the feature matrix to be processed corresponding to each frame image matrix, a sequence of feature matrices to be processed is determined.

[0155] Optionally, the processor 301 executes the step of acquiring a sequence of image matrices of a plurality of containers to be inspected within a preset time, including:

[0156] Within a preset time, a plurality of containers to be inspected are periodically photographed in a horizontal direction by means of sliding photographing equipment installed at different heights on the lashing bridge to obtain a sequence of original photographed images of each container to be inspected;

[0157] Determining a target image of each container to be inspected in each shooting cycle according to a sequence of original captured images of each container to be inspected;

[0158] The image matrix sequence is determined according to the target image of each container to be inspected in each shooting cycle and the adjacency relationship between the containers to be inspected.

[0159] Optionally, the processor 301 performs image feature extraction processing on each target image in each frame image matrix to obtain image feature values ​​of each target image in each frame image matrix, including:

[0160] Performing image feature extraction processing on each of the target images using a trained feature extraction model to obtain at least one of an offset feature, an angle feature, and a lashing feature of the container to be detected in each of the target images;

[0161] An image feature value of each target image in each frame image matrix is ​​determined according to at least one of the offset feature, the angle feature, and the binding feature corresponding to each target image.

[0162] Optionally, the processor 301 performs the first feature extraction process on each frame of the feature matrix to be processed in the feature matrix sequence to be processed to obtain the spatial feature vector of the feature matrix sequence to be processed, including:

[0163] Performing convolution processing on the feature matrix to be processed in each frame to obtain spatial features of the feature matrix to be processed in each frame;

[0164] Expanding the spatial features of the feature matrix to be processed in each frame in one dimension to obtain expanded spatial features;

[0165] The expanded spatial features are spliced ​​in the time dimension to obtain the spatial feature vectors of the feature matrix sequence to be processed.

[0166] Optionally, the processor 301 performs a second feature extraction process on the spatial feature vector to obtain the spatiotemporal feature vector of the feature matrix sequence to be processed, including:

[0167] Splitting the spatial feature vector according to a preset length to obtain a plurality of spatial feature sub-vectors;

[0168] The spatiotemporal feature vectors of the feature matrix sequence to be processed are obtained by performing a second feature extraction process on the plurality of spatial feature sub-vectors.

[0169] Optionally, the processor 301 performs the second feature extraction process using the plurality of spatial feature subvectors to obtain the spatiotemporal feature vectors of the feature matrix sequence to be processed, including:

[0170] Mapping each of the spatial feature subvectors into respective preset spaces through a preset mapping path to obtain a target vector space, wherein the target vector space includes a target feature subvector obtained by mapping the spatial feature subvectors, each mapping path is used to map one of the spatial feature subvectors into one of the target feature subvectors, and one target vector space corresponds to one matrix unit in the feature matrix to be processed;

[0171] Performing weighted fusion on all target feature subvectors in each of the target vector spaces to obtain fused feature vectors corresponding to each of the target vector spaces;

[0172] Performing a nonlinear transformation on the fused feature vector to obtain a spatiotemporal feature vector corresponding to the target vector space;

[0173] The spatiotemporal eigenvectors of the feature matrix sequence to be processed are determined according to the spatiotemporal eigenvectors corresponding to the target vector space.

[0174] It should be noted that the electronic device provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can perform the container detection method.

[0175] The electronic device provided in the embodiment of the present invention can implement each process of the container detection method in the above method embodiment and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0176] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the various processes of the container detection method or the application-end container detection method provided in the embodiment of the present invention, and can achieve the same technical effect. To avoid repetition, the details are not described here.

[0177] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0178] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A container detection method, characterized in that: The method comprises the following steps: When the number of containers to be inspected that are adjacent to each other is greater than a preset number, within a preset time, a plurality of containers to be inspected are periodically photographed in a horizontal direction by means of sliding photographing devices installed at different heights on the lashing bridge to obtain a sequence of original photographed images of each container to be inspected; based on the sequence of original photographed images of each container to be inspected, a target image of each container to be inspected in each photographing cycle is determined; based on the target image of each container to be inspected in each photographing cycle and the adjacent relationship between each container to be inspected, an image matrix sequence is determined, wherein the image matrix sequence includes image matrices arranged according to the time dimension. The image matrix includes a plurality of target images of the containers to be inspected arranged in an adjacency relationship; image feature extraction processing is performed on each target image in each frame of the image matrix to obtain image feature values ​​of each target image in each frame of the image matrix; based on the image feature values ​​of each target image in each frame of the image matrix, a feature matrix to be processed corresponding to each frame of the image matrix is ​​determined; based on the feature matrix to be processed corresponding to each frame of the image matrix, a feature matrix sequence to be processed is determined, the feature matrix sequence to be processed including a feature matrix to be processed arranged in a time dimension, the feature matrix to be processed including the image feature values ​​of the plurality of containers to be inspected arranged in an adjacency relationship; Performing a first feature extraction process on each frame of the feature matrix to be processed in the feature matrix sequence to be processed to obtain a spatial feature vector of the feature matrix sequence to be processed; Performing a second feature extraction process on the spatial feature vector to obtain a spatiotemporal feature vector of the feature matrix sequence to be processed; The detection results of the plurality of containers to be detected are determined according to the spatiotemporal feature vectors.

2. The container detection method according to claim 1, wherein: The performing image feature extraction processing on each target image in each frame image matrix to obtain the image feature value of each target image in each frame image matrix includes: Performing image feature extraction processing on each of the target images using a trained feature extraction model to obtain at least one of an offset feature, an angle feature, and a lashing feature of the container to be detected in each of the target images; An image feature value of each target image in each frame image matrix is ​​determined according to at least one of the offset feature, the angle feature, and the binding feature corresponding to each target image.

3. The container detection method according to any one of claims 1 to 2, characterized in that: The step of performing a first feature extraction process on each frame of the feature matrix sequence to be processed to obtain a spatial feature vector of the feature matrix sequence to be processed includes: Performing convolution processing on the feature matrix to be processed in each frame to obtain spatial features of the feature matrix to be processed in each frame; Expanding the spatial features of the feature matrix to be processed in each frame in one dimension to obtain expanded spatial features; The expanded spatial features are spliced ​​in the time dimension to obtain the spatial feature vectors of the feature matrix sequence to be processed.

4. The container detection method according to claim 3, characterized in that: The performing a second feature extraction process on the spatial feature vector to obtain the spatiotemporal feature vector of the feature matrix sequence to be processed includes: Splitting the spatial feature vector according to a preset length to obtain a plurality of spatial feature sub-vectors; The spatiotemporal feature vectors of the feature matrix sequence to be processed are obtained by performing a second feature extraction process on the plurality of spatial feature sub-vectors.

5. The container detection method according to claim 4, characterized in that: The performing of a second feature extraction process on the plurality of spatial feature sub-vectors to obtain the spatiotemporal feature vectors of the feature matrix sequence to be processed includes: Mapping each of the spatial feature subvectors into respective preset spaces through a preset mapping path to obtain a target vector space, wherein the target vector space includes a target feature subvector obtained by mapping the spatial feature subvectors, each mapping path is used to map one of the spatial feature subvectors into one of the target feature subvectors, and one target vector space corresponds to one matrix unit in the feature matrix to be processed; Performing weighted fusion on all target feature subvectors in each of the target vector spaces to obtain fused feature vectors corresponding to each of the target vector spaces; Performing a nonlinear transformation on the fused feature vector to obtain a spatiotemporal feature vector corresponding to the target vector space; The spatiotemporal eigenvectors of the feature matrix sequence to be processed are determined according to the spatiotemporal eigenvectors corresponding to the target vector space.

6. A container detection device, characterized in that: The container detection device comprises: The acquisition module is configured to, when the number of adjacent containers to be inspected is greater than a preset number, periodically slide photograph the plurality of containers to be inspected in the horizontal direction within a preset time using sliding photographing devices installed at different heights on the lashing bridge to obtain a sequence of original photographed images of each container to be inspected; determine the target image of each container to be inspected in each photographing cycle based on the sequence of original photographed images of each container to be inspected; determine an image matrix sequence based on the target image of each container to be inspected in each photographing cycle and the adjacency relationship between the containers to be inspected, the image matrix sequence including images arranged in a time dimension A matrix is ​​prepared, wherein the image matrix includes a plurality of target images of the containers to be inspected arranged in an adjacency relationship; image feature extraction processing is performed on each target image in each frame of the image matrix to obtain an image feature value of each target image in each frame of the image matrix; based on the image feature value of each target image in each frame of the image matrix, a feature matrix to be processed corresponding to each frame of the image matrix is ​​determined; based on the feature matrix to be processed corresponding to each frame of the image matrix, a feature matrix sequence to be processed is determined, wherein the feature matrix sequence to be processed includes a feature matrix to be processed arranged in a time dimension, and the feature matrix to be processed includes the image feature values ​​of the plurality of containers to be inspected arranged in an adjacency relationship; A first processing module is configured to perform a first feature extraction process on each frame of the feature matrix to be processed in the feature matrix sequence to be processed, so as to obtain a spatial feature vector of the feature matrix sequence to be processed; A second processing module is used to perform a second feature extraction process on the spatial feature vector to obtain the spatiotemporal feature vector of the feature matrix sequence to be processed; A determination module is used to determine the detection results of the multiple containers to be detected according to the spatiotemporal feature vector.

7. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the container detection method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the container detection method according to any one of claims 1 to 5 are implemented.

Images