Intelligent work station image and customs declaration data association method based on edge computing

Abstract

This invention relates to the field of image processing technology, specifically to a method for associating inspection images and customs declaration data at intelligent workstations based on edge computing. Addressing the problem of low inspection efficiency due to insufficient semantic segmentation accuracy in existing technologies, this invention uses foreground region features as a reference to dynamically repair and match the initially segmented semantic region through superpixel blocks, adaptively selecting the optimal feature dimension, and finally associating the optimized target semantic region with the customs declaration data. This method effectively improves the accuracy of image segmentation and enhances the query efficiency and reliability after associating inspection images with customs declarations.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for associating intelligent workstation inspection images with customs declaration data based on edge computing. Background Technology

[0002] Directly linking customs declaration data with workstation inspection images presents several challenges. First, the large volume of data leads to slow loading, long query times, and low efficiency when retrieving images associated with customs declaration data. Second, it's impossible to directly retrieve the items inside the cargo container from images associated with customs declaration data, or vice versa. A common approach is to use a semantic segmentation neural network model to segment the semantic regions of all images and then associate these regions with the customs declaration data. However, due to the diverse types and characteristics of items inside cargo containers, the training samples for the semantic segmentation neural network model cannot cover a large number of items (or the cost of covering a large number of items is too high). This results in insufficient generalization ability of the semantic segmentation neural network model, leading to poor accuracy in the segmented semantic regions. Consequently, after associating with customs declaration data, the efficiency and accuracy of retrieving items based on customs declaration data, or vice versa, are low, and the query results are unsatisfactory. Summary of the Invention

[0003] To address the problem that inaccurate semantic regions segmented by semantic segmentation neural network models lead to low query efficiency and unreliability after association with customs declaration data, this invention provides a method for associating intelligent workstation inspection images with customs declaration data based on edge computing.

[0004] The method for associating intelligent workstation inspection images and customs declaration data based on edge computing in this invention adopts the following technical solution: One embodiment of the present invention provides a method for associating inspection images and customs declaration data at intelligent workstations based on edge computing. The method includes the following steps: For all frame images collected during the cargo inspection process, the foreground region of each frame image is obtained, and the first reference feature of the foreground region of each frame image is extracted respectively. Each frame image is divided into several superpixel blocks, and each semantic region in each frame image is repaired using the superpixel blocks. The feature matching degree between the second reference feature of each repaired semantic region and all the first reference features in several preset dimensions is obtained. The difference between the repaired semantic region and the foreground region when the feature matching degree is maximized is recorded as the correction error in several dimensions. The several dimensions with the smallest correction error are recorded as the target dimension. When the second reference feature of each repaired semantic region has the highest feature matching degree with all first reference features in the target dimension, the repaired semantic region is recorded as each target semantic region in each frame image; the target semantic regions of all frames images are then associated with the customs declaration data.

[0005] Preferably, the specific steps of using superpixel blocks to repair each semantic region in each frame of the image and obtaining the feature matching degree between the second reference feature of each repaired semantic region and all the first reference features in a preset number of dimensions are as follows: S1: The superpixel blocks through which the contour of each semantic region in each frame of an image passes are denoted as the reference block; S2: In all reference blocks of each semantic region, randomly select several neighboring superpixel blocks of each reference block to adjust the semantic region, and obtain the repaired semantic region corresponding to each semantic region; for the first reference feature obtained in any frame image, calculate the similarity between the first reference feature and the second reference feature of the repaired semantic region in several preset dimensions; the maximum value of the obtained similarity for the first reference feature of all frames image is taken as the feature matching degree.

[0006] Preferably, the repaired semantic region obtained when the feature matching degree is maximized refers to the repaired semantic region obtained when the feature matching degree is maximized among all the feature matching degrees obtained after repeatedly executing S2 several times.

[0007] Preferably, the difference between the repaired semantic region and the foreground region obtained when the feature matching degree is maximized is denoted as a correction error in several dimensions, and the specific steps include the following: The repaired semantic region obtained when the feature matching degree is maximized under each semantic region is recorded as the first region of each semantic region. In the first region of all semantic regions in each frame image, the first region with the largest intersection-union ratio (IU) with the foreground region of each frame image is obtained and denoted as the target inpainting region; the IU of the target inpainting region and the foreground region is denoted as the first matching coefficient of each frame image; the correction error of the several dimensions is negatively correlated with the mean of the first matching coefficient of all frame images.

[0008] Preferably, the dimensions in which the correction error is minimized are denoted as the target dimensions, and the specific steps include the following: Randomly select several dimensions from the first reference features and denote them as candidate dimensions, and obtain the correction error of the candidate dimensions; for all selection results of candidate dimensions, among the candidate dimensions corresponding to all selection results, denote the candidate dimension with the smallest correction error as the target dimension.

[0009] Preferably, the specific steps involved in associating the target semantic regions of all frame images with customs declaration data are as follows: For all target semantic regions in all frame images, and for all target semantic regions corresponding to second reference features, all second reference features are clustered to obtain several categories. The mean of the second reference features in each category is denoted as the category center. The second reference feature with the greatest similarity to the category center in each category, and several second reference features with the least similarity to the category center are denoted as the target reference features of each category. A data table is created in the database. Each row of the data table stores data including customs declaration data, target reference features of all categories, and the storage path of the video composed of all frame images.

[0010] Preferably, the specific steps of randomly selecting several neighboring superpixel blocks of each reference block to adjust the semantic region to obtain the repaired semantic region corresponding to each semantic region are as follows: For each semantic region in each frame of the image, all reference blocks are traversed. For any reference block that is traversed, several neighboring superpixel blocks are randomly expanded from the reference block to the inside or outside of the semantic region. Among them, the superpixel blocks expanded to the inside of the semantic region are called the inward pruning region, and the superpixel blocks expanded to the outside of the semantic region are called the outward pruning region. The first trimmed region is obtained by taking the union of each semantic region with the outward trimmed regions of all reference blocks. The repaired semantic region is obtained by taking the difference between the first trimmed region and the inward trimmed regions of all reference blocks.

[0011] Preferably, the specific steps for calculating the similarity between the first reference feature and the second reference feature of the repaired semantic region across several preset dimensions are as follows: For any one of the preset dimensions, obtain the cosine similarity between the feature vector of the first reference feature in that dimension and the feature vector of the second reference feature in that dimension. The mean of the cosine similarities obtained for each of the preset dimensions is denoted as the similarity under the preset dimensions.

[0012] Preferably, the specific steps for obtaining the first reference feature, the second reference feature, and the feature vector are as follows: For pixels in the foreground region or the repaired semantic region, the feature map output by the encoder is used as the first reference feature or the second reference feature; each channel in the feature map is regarded as each dimension of the first reference feature or the second reference feature; the row vector formed by the gray values ​​of each channel in the feature map is regarded as the feature vector of the first reference feature or the second reference feature in each dimension.

[0013] Preferably, the loss function used when training the autoencoder consists of a mean squared error loss function and a feature extraction loss function, wherein the method for obtaining the feature extraction loss function is as follows: The cosine similarity between the feature vector of each channel of the feature map and the feature vectors of other channels is calculated and the average value is taken. This average value is denoted as the similarity feature of each channel. The mean of the squares of the similarity features of all channels is used as the feature extraction loss function.

[0014] The beneficial effects of the technical solution of the present invention are: This invention creatively introduces a foreground region feature-guided superpixel inpainting mechanism. The mechanism works as follows: this method does not rely entirely on the initial segmentation result, but instead uses the features of the actual object being inspected (the foreground region) as anchor points. Superpixel blocks are used to inpaint the initially segmented semantic region, and the matching degree between the inpainted region and the foreground region features is calculated in multiple dimensions. Finally, the inpainted result with the highest matching degree is selected as the target semantic region. This method further effectively overcomes the inherent deficiency of insufficient generalization ability in semantic segmentation models. Even if the initial segmented region has deviations (such as inaccurate boundaries, inclusion of background, or omission of some objects), the subsequent inpainting and matching processes can greatly optimize and correct the segmentation result, thereby obtaining a more accurate and reliable single-object image region.

[0015] Furthermore, this invention avoids matching on invalid or redundant feature dimensions by introducing an adaptive filtering mechanism for target dimensions. This mechanism does not blindly match across all feature dimensions, but rather automatically identifies the dimensions (i.e., target dimensions) that best distinguish and characterize the key features of items in the current inspection scenario by calculating correction errors under different dimension combinations. The final matching and region determination are performed in this optimized dimensional space. This mechanism ensures the efficiency and specificity of feature matching. It can adapt to different cargo types and inspection environments, automatically focusing on the most discriminative features, thereby reducing noise interference and improving the accuracy and efficiency of subsequent data association.

[0016] Furthermore, this invention directly improves the quality and usability of associating customs declaration data with image content. Since the association is based on target semantic regions optimized through the aforementioned process, these regions accurately represent individual items within the container. Therefore, the established associations are of high quality and high confidence.

[0017] In summary, this invention improves the efficiency and reliability of querying data after associating workstation inspection images with customs declaration data by improving segmentation accuracy, optimizing feature selection, and ultimately ensuring the quality of association. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the steps of a method for associating inspection images and customs declaration data at an intelligent workstation based on edge computing, as provided in an embodiment of the present invention. Detailed Implementation

[0020] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of the edge computing-based intelligent workstation image and customs declaration data association method proposed by the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] The following description, in conjunction with the accompanying drawings, details the specific scheme of the method for associating intelligent workstation inspection images and customs declaration data based on edge computing provided by the present invention.

[0023] Please see Figure 1 The diagram illustrates a flowchart of a method for associating inspection images and customs declaration data at an intelligent workstation based on edge computing, according to an embodiment of the present invention. The method includes the following steps: Step S101: For all frame images collected during the cargo inspection process, obtain the foreground region of each frame image.

[0024] The cargo inspection process refers to the process of opening the customs container and manually inspecting the contents of the container; during the cargo inspection process, the manual inspection process is captured by a camera, and all frames of images are obtained, which are the workstation inspection images.

[0025] The foreground region refers to the connected components of an object that moves during the goods inspection process within each frame of the image. It should be noted that during goods inspection, the inspector manually inspects one item at a time from multiple angles (i.e., multiple viewpoints). In this case, the moving object refers to the item being inspected. Furthermore, since the inspector (including their hand handling the items) is also in motion, the foreground region in this embodiment does not include the pixel area where the inspector is located.

[0026] As an example, the method for obtaining the foreground region is as follows: All frame images are input into a Gaussian mixture background modeling algorithm to obtain the initial foreground region of each frame. Further, the semantic region of the inspector in each frame is segmented, and pixel regions that are in the initial foreground region but not in the semantic region are recorded as the foreground region.

[0027] It should be noted that each frame of this embodiment is a color image (i.e., a three-channel RGB image). In this embodiment, a semantic segmentation neural network model is used to segment the semantic region of the inspector in each frame of the image.

[0028] Step S102: Extract the first reference features of the foreground region of each frame image.

[0029] The foreground region of each frame represents the area in which a single item inside the container is presented in the image. The first reference feature refers to the image features of the item in different dimensions in each frame, such as image features in dimensions such as color and texture.

[0030] The purpose of obtaining the first reference feature in this embodiment is to describe the common characteristics of the same type of items in the cargo box, considering that there are generally multiple items of the same model, size and material (hereinafter referred to as items of the same model) in the cargo box by extracting the first reference feature; and to provide a data basis for segmenting other item regions in each frame of the image in the subsequent process.

[0031] For all the first reference features corresponding to the foreground region of all frames of images, the set of these first reference features represents the image features of multiple items taken out by the inspectors during the entire inspection process, as well as the image features of an item under the observation views of multiple inspectors.

[0032] Step S103: Divide each frame image into several superpixel blocks, and use the superpixel blocks to repair each semantic region in each frame image, and obtain the feature matching degree between the second reference feature of each repaired semantic region and all the first reference features in several preset dimensions; the difference between the repaired semantic region and the foreground region when the feature matching degree is the maximum is recorded as the correction error in several dimensions.

[0033] The superpixel block refers to a region composed of several pixels. Each superpixel block represents a region in the image with relatively uniform color distribution that can represent local details of objects in the image.

[0034] As an example, each frame of an image is divided into several superpixel blocks. The method includes: using the SLIC algorithm (Superpixel Segmentation Algorithm) to divide each frame of an image into several superpixel blocks; in the SLIC algorithm, the number of superpixels is set to 5000, the compactness factor is set to 10, and the number of generations is set to 10; in other embodiments, other algorithms can also be used to obtain superpixel blocks, or the parameters of the SLIC algorithm can be set to other values. This embodiment does not impose specific limitations.

[0035] This step utilizes superpixel blocks to repair the semantic regions of each frame of the image. The purpose is to address the limitation of existing AI-based segmentation algorithms, such as semantic segmentation neural network models, which often fail to accurately segment the regions of all items (i.e., the semantic regions) in each frame. This is because the types of items in the cargo container are diverse, with significant differences in their features. The training samples of the semantic segmentation neural network model cannot cover a large number of items (or the cost of covering a large number of items is too high), resulting in insufficient generalization ability and consequently, poor accuracy in segmenting the semantic regions. Superpixel blocks represent local pixel regions of an item, with each superpixel block located on the same item. By using some superpixel blocks to repair the semantic regions obtained by the semantic segmentation neural network model, the semantic regions of each item (items outside the foreground region) in the image can be reconstructed. In other words, some superpixel blocks can fine-tune the boundary contours of the semantic regions, making the repaired semantic regions more accurate.

[0036] It should be noted that the semantic segmentation neural network model described in this step is the same model as the semantic segmentation neural network model described in step S101. As an example, DeepLabV3 is used as the semantic segmentation neural network model, and other semantic segmentation neural network models can also be used in other embodiments. The semantic segmentation neural network model is existing technology, and this embodiment will not elaborate on or limit it.

[0037] During the repair process, the second reference feature of the repaired semantic region refers to the image features of the items in the repaired semantic region in different dimensions; the second reference feature and the first reference feature have the same meaning and acquisition process, the only difference being that the former is for the repaired semantic region, while the latter is for the foreground region.

[0038] The feature matching degree between the second reference feature and the first reference feature of the repaired semantic region in several preset dimensions indicates whether the item corresponding to the repaired semantic region has the same or similar characteristics as the item taken out by the inspector (i.e., the item corresponding to the foreground region) in the preset dimensions. The larger the feature matching degree, the more similar the characteristics are.

[0039] The repaired semantic region obtained when the feature matching degree is maximized has the most identical or similar features to the items in the foreground region, indicating that the repaired semantic region is the optimal reconstruction or fine-tuning result of the semantic region directly obtained by the semantic segmentation neural network model.

[0040] This embodiment further considers that, due to the uncertainty regarding whether the first or second reference features contain redundant or interfering features, it is impossible to determine in which dimensions the item taken out by the inspector (i.e., the item corresponding to the foreground area) shares the same or similar characteristics with other items of the same model in the box. Based on this, this embodiment utilizes the correction error to describe whether the item taken out by the inspector (i.e., the item corresponding to the foreground area) shares the same or similar characteristics with other items of the same model in the box under specific dimensions. A larger correction error indicates a greater difference between the repaired semantic region and the foreground region, suggesting that the repaired semantic region may be inaccurate under specific dimensions (i.e., the preset dimensions). This indicates that the dimensions selected in obtaining the repaired semantic region are inappropriate, and that items do not share the same or similar characteristics under these dimensions.

[0041] Step S104: When the correction error is minimized, several dimensions are recorded as target dimensions; when the feature matching degree between the second reference feature and the first reference feature of each repaired semantic region is maximized in the target dimension, the repaired semantic region is recorded as each target semantic region in each frame image.

[0042] The target dimension refers to the dimension when the item taken out by the inspector (i.e. the item corresponding to the foreground area) has the same or similar characteristics as other items of the same type in the box, which are also the dimensions that minimize the correction error.

[0043] When the second reference feature of the repaired semantic region has the highest feature matching degree with the first reference feature in the target dimension, it indicates that the semantic region has been repaired most accurately and reliably. The repaired semantic region obtained at this time is recorded as the target semantic region corresponding to each semantic region in each frame image.

[0044] Step S105: Associate the target semantic regions of all frame images with the customs declaration data.

[0045] Current technologies typically associate customs declaration data with a video composed of all frame images. However, this approach suffers from several drawbacks. First, the sheer volume of frame images leads to slow loading, long query times, and low efficiency when querying all associated frame images based on customs declaration data. Second, it's impossible to directly retrieve the items inside the cargo container from all frame images associated with customs declaration data, or vice versa. A common approach to address this is to use a semantic segmentation neural network model to directly segment the semantic regions of all frame images and then associate these regions with the customs declaration data. However, the semantic regions segmented directly by the semantic segmentation neural network model are often inaccurate. For example, the semantic regions may not contain the complete outline of a single item or may contain the outlines of other items. Consequently, after associating with the customs declaration data, the efficiency and accuracy of retrieving items based on the customs declaration data, or retrieving customs declaration data based on items inside the cargo container, are low, resulting in unsatisfactory query results.

[0046] In this embodiment, by repairing the semantic region directly segmented by the semantic segmentation neural network model, the target semantic region is made more accurate. After associating the target semantic region with customs declaration data, the efficiency or accuracy of the associated data query is improved because the target semantic region can accurately describe a single item, making the query based on the association result more reliable, practical and efficient.

[0047] As a preferred example, the first reference features of the foreground region are extracted from all frames of the image, including the following methods: The pixels within the foreground region of each frame are input into the trained autoencoder. The multi-channel feature map output by the autoencoder serves as the first reference feature for the foreground region, where each channel of the feature map is considered as each dimension of the first reference feature. The row vector formed by flattening the grayscale values ​​of each channel of the feature map is considered as the feature vector of the first reference feature in each dimension.

[0048] Specifically, the pixels within the foreground region of each frame are input into the trained autoencoder. The method is as follows: for each frame's foreground region bounding box (obtained using the AABB bounding box algorithm), all pixels within the bounding box constitute a sub-image, which is then input into the trained autoencoder. It should be noted that the pixel values ​​of pixels outside the foreground region in the sub-image are set to (0, 0, 0).

[0049] As an example, the first reference feature contains 32 dimensions, that is, the feature map has 32 channels.

[0050] As a preferred example, superpixel blocks are used to inpaint each semantic region in each frame of the image, and the feature matching degree between the second reference feature of each inpainted semantic region and all the first reference features in several preset dimensions is obtained. The method includes: (1) For each semantic region in each frame of the image, all superpixel blocks that the contour of the semantic region passes through (or traverses) are denoted as reference blocks. In particular, if the contour of the semantic region happens to pass through the common boundary of two superpixel blocks, in this embodiment, the superpixel block that is outside the semantic region is taken as the reference block.

[0051] (2) In all the reference blocks of each semantic region, randomly select several neighboring superpixel blocks of each reference block to adjust the semantic region, and obtain the repaired semantic region corresponding to each semantic region.

[0052] For any frame of image, the similarity between the first reference feature and the second reference feature of the repaired semantic region is calculated in several preset dimensions. For all first reference features obtained from all frames of image, the maximum value of the obtained similarity for each first reference feature is taken as the feature matching degree.

[0053] All of the first reference features represent the image features of multiple items taken out by the inspectors during the entire inspection process, as well as the image features of a single item under different inspection perspectives. The above process retrieves the image features that best match the repaired semantic region from the image features of multiple items and the image features of a single item under different inspection perspectives. The similarity between the image features and the image features of the repaired semantic region is represented as the feature matching degree. The greater the feature matching degree, the more reliable the repaired semantic region obtained by the above adjustment is.

[0054] The second reference feature of the repaired semantic region is obtained in the same way as the first reference feature of the foreground region, that is, the pixels in the repaired semantic region of each frame image are output to the autoencoder to obtain the second reference feature; this embodiment will not be described in detail.

[0055] It should be noted that some models of items may never have been picked up by the inspectors during the entire inspection process. That is, not all models of items in the cargo box may be included in the foreground areas of all frames of images. This suggests that the following situation may exist: for some semantic regions in each frame of image, the items corresponding to the semantic regions are different from any items corresponding to any of the foreground areas. In this case, the feature matching degree obtained is small.

[0056] As a preferred example, several neighboring superpixel blocks of each reference block are randomly selected to adjust the semantic region, resulting in the repaired semantic region corresponding to each semantic region. The methods include: For each semantic region in each frame of the image, traverse all reference blocks. For any reference block encountered, randomly generate an integer n. Randomly expand |n| superpixel blocks into the neighborhood of this reference block. Here, n < 0 indicates expansion only into the semantic region, and n > 0 indicates expansion only into the semantic region. The connected component formed by all the expanded superpixel blocks is denoted as the pruned region of this reference block. The pruned region obtained when n < 0 is denoted as the inward pruned region, and n > 0 indicates the outward pruned region. Specifically, when n = 0, no expansion is performed, and the reference block is randomly labeled as either an inward pruned region or an outward pruned region.

[0057] After traversing all the reference blocks, a pruned region or an inward pruned region is randomly generated for each reference block. The union of each semantic region with the pruned regions of all reference blocks is obtained to get the first pruned region. The difference between the first pruned region and the inward pruned regions of all reference blocks is obtained to get the repaired semantic region.

[0058] As an example, randomly expanding |n| superpixel blocks from the base block to the neighborhood can be achieved using the following methods: Randomly select a superpixel block that is adjacent to the reference block (i.e., shares a common boundary), and denot it as the 1-neighbor superpixel block. Randomly select a superpixel block that is adjacent to the 1-neighbor superpixel block (excluding any reference block), and denot it as the 2-neighbor superpixel block. Randomly select a superpixel block that is adjacent to the 2-neighbor superpixel block (excluding any reference block and the 1-neighbor superpixel block), and denot it as the 3-neighbor superpixel block. Randomly select a superpixel block that is adjacent to the 3-neighbor superpixel block (excluding any reference block and the 1- and 2-neighbor superpixel blocks), and denot it as the 4-neighbor superpixel block. And so on to obtain the |n|-neighbor superpixel blocks.

[0059] The connected region consisting of the base block, the 1-neighbor superpixel block, the 2-neighbor superpixel block, ..., the |n|-neighbor superpixel block is denoted as the expanded n superpixel blocks.

[0060] It should be noted that if a certain neighboring superpixel block is missing, for example, if the 3-neighboring superpixel block is missing, then subsequent neighboring superpixel blocks will not be acquired. For example, the 3, 4, ..., |n| neighboring superpixel blocks will not be acquired. Only the connected component formed by the base block and the 1 and 2 neighboring superpixel blocks will be counted as the expanded |n| superpixel blocks. It should also be noted that when n is less than 0, only neighboring superpixel blocks within the semantic region are acquired; when n is greater than 0, only neighboring superpixel blocks outside the semantic region are acquired.

[0061] As an example, the random number n refers to an integer in the range [-2, 2].

[0062] As a preferred example, calculating the similarity between the first reference feature and the second reference feature of the repaired semantic region across several preset dimensions includes the following steps: For any one of the preset dimensions, obtain the cosine similarity between the feature vector of the first reference feature in that dimension and the feature vector of the second reference feature in that dimension. The mean of the cosine similarities obtained for each of the preset dimensions is denoted as the similarity under the preset dimensions.

[0063] As a preferred example, the method for obtaining the repaired semantic region when the feature matching degree is maximized is as follows: Repeat step (2) above N0 times. Each time step (2) is executed, some superpixel blocks are randomly selected to trim or adjust the boundaries of the semantic region. After repeating step (2) above several times, among all the obtained feature matching degrees, only the repaired semantic region obtained when the feature matching degree is the largest is retained.

[0064] N0 is a preset value. In one embodiment, N0 is set as follows: Get the absolute value of the upper limit and the absolute value of the lower limit of the range of n. The maximum value of the two absolute values ​​is denoted as n1. n1 represents the maximum number of expanded neighbors when expanding the neighborhood superpixel block.

[0065] For each reference block, obtain the total number of its n1-neighbor superpixel blocks (including superpixel blocks within and outside the semantic region). The sum of the total number of 1-neighbor superpixel blocks, 2-neighbor superpixel blocks, ..., n1-neighbor superpixel blocks is recorded as the total number of n-neighbor superpixel blocks for each reference block. For all reference blocks traversed by each reference block, a multiple of the average of the total number of all reference blocks is taken as N0 (rounded up). In one example, to save computation, 5 times the average is taken as N0 (rounded up); in another example, to further ensure the accuracy of the repaired semantic region, 7 times the average is taken as N0 (rounded up).

[0066] Thus, for each semantic region in each frame of the image, the repaired semantic region with the maximum feature matching degree is obtained using the above method.

[0067] It should also be noted that in step S104, when the second reference feature of the repaired semantic region has the maximum feature matching degree with all the first reference features in the target dimension, the repaired semantic region is recorded as the target semantic region of each frame image. This process is similar to the above process, that is, after repeating step (2) N0 times, among all the obtained feature matching degrees, only the repaired semantic region obtained when the feature matching degree is the maximum is retained, and recorded as the target semantic region corresponding to each semantic region in each frame image. It should be noted that when repeating step (2) and obtaining the feature matching degree, the feature matching degree is obtained based on the target dimension, not based on several preset dimensions.

[0068] It should be further noted that, as mentioned above, the following situation may occur: for certain semantic regions within each frame of the image, the item corresponding to that semantic region is different from any item model corresponding to any foreground region. In this case, the semantic region cannot be repaired using superpixel blocks based on the first reference feature as described above. Specifically, for any semantic region in each frame of the image, and for the feature matching degree value that maximizes the feature matching degree corresponding to that semantic region, if the value is less than the first threshold th1, then the semantic region is not repaired, and the target semantic region corresponding to that semantic region is the semantic region directly output by the semantic segmentation neural network model, without needing to obtain the target semantic region corresponding to that semantic region according to step S104. If it is greater than or equal to the first threshold th1, then the target semantic region corresponding to that semantic region is obtained according to step S104. This embodiment uses th1=0.65 as an example for description.

[0069] As a preferred example, the difference between the repaired semantic region and the foreground region obtained when the feature matching degree is maximized is denoted as the correction error in several dimensions, including the following methods: For each semantic region in each frame of the image, the repaired semantic region with the maximum feature matching degree is obtained using the above method. The repaired semantic region obtained with the maximum feature matching degree is referred to as the first region of each semantic region.

[0070] In each frame of an image, the first region with the largest intersection-union ratio (IUU) with the foreground region is selected from all semantic regions. This region is denoted as the target inpainting region. The target inpainting region represents the inpainted semantic region of the item obtained by the semantic segmentation neural network model for the item corresponding to the foreground region. The IUU ratio (i.e., the ratio of the area of ​​the intersection to the area of ​​the union) between the target inpainting region and the foreground region is denoted as the first matching coefficient of each frame. The larger the first matching coefficient, the higher the overlap between the target inpainting region and the foreground region.

[0071] Let y be the mean of the first matching coefficients of all frames. Let exp(-y) be the difference between the repaired semantic region and the foreground region when the feature matching degree is maximized, which is also the correction error in several dimensions. The larger the first matching coefficient of all frames, the higher the overlap between the target repaired region and the foreground region, and the smaller the correction error.

[0072] As a preferred example, the dimensions for minimizing the correction error are denoted as the target dimensions, including the following methods: Several dimensions are randomly selected from the first reference features and denoted as candidate dimensions. The correction error of the candidate dimensions is then obtained again according to the method in step S103. Here, the preset number of dimensions mentioned in step S103 refers to several dimensions randomly selected from the first reference features.

[0073] For all selection results of candidate dimensions (including candidate dimensions composed of several preset dimensions), each selection result corresponds to a correction error. The selection result with the smallest correction error among all correction errors, that is, the corresponding candidate dimension, is denoted as the target dimension.

[0074] As an example, the number of candidate dimensions selected each time is set to a fixed value, such as selecting 65% of the dimensions (rounded up) of the first reference feature each time as candidate dimensions.

[0075] As a preferred example, the method of associating the target semantic regions of all frame images with customs declaration data includes: For all target semantic regions in all frame images, and for all corresponding second reference features, all second reference features are clustered to obtain several categories. The second reference features in each category are identical or similar, representing items of the same type. The mean of the second reference features in each category is denoted as the category center. The second reference feature with the highest similarity to the category center and the n² second reference features with the lowest similarity to the category center are all denoted as the target reference features of each category. In this embodiment, n² is set to 40% (rounded up) of the number of second reference features within a category.

[0076] The purpose of using the second reference feature with the greatest similarity as the target reference feature is to extract the most representative items of the same type and associate them with the customs declaration data. The purpose of using multiple second reference features with the least similarity as the target reference features is to extract items of the same type that are significantly different from the representative items and associate them with the customs declaration data. Ultimately, this results in a certain diversity of items associated with the customs declaration data, but it is not necessary to associate all items.

[0077] At this point, all target reference features have been obtained for all frame images.

[0078] Create a data table in the database. Each row of the data table stores data including the customs declaration number, import / export port, commodity name, and quantity of goods, as well as all target reference features and the server-side storage path of the video composed of all frame images.

[0079] In this example, not only is customs declaration data associated with video data, but it is also associated with the target semantic regions of all frames in the video. This avoids the inefficiency of directly accessing or loading video data when querying based on customs declaration data. This example only requires inputting the target reference features from the data table into the decoder of the autoencoder to reconstruct the item image, resulting in high query efficiency based on customs declaration data. Furthermore, when querying customs declaration data based on a user-provided item image, it only requires inputting the item image into the encoder of the autoencoder to obtain its second reference feature, and then querying the customs declaration data corresponding to the target reference feature with the highest similarity to the second reference feature in the data table. More importantly, the target reference features obtained in this embodiment are obtained from the semantic regions that have undergone the aforementioned repair. Compared to semantic regions obtained directly based on the semantic segmentation neural network model, the associated query effect of this embodiment is more efficient and accurate.

[0080] As an example, clustering all the second reference features yields several categories, including: The feature vectors of all dimensions in each second reference feature are concatenated end to end to obtain a concatenated vector. The cosine similarity of the concatenated vectors of any two second reference features is denoted as the similarity x between the two second reference features. exp(-x) is used as the cluster distance between any two second reference features. exp() represents an exponential function with the natural constant as the base. The larger the similarity x between any two second reference features, the smaller the cluster distance between them.

[0081] Based on this clustering distance, the mean-shift clustering algorithm is used to perform clustering to obtain all categories.

[0082] In this embodiment, the bandwidth of the mean-shift clustering algorithm is set to 0.1. This algorithm is a well-known technique, and will not be described in detail in this embodiment. It should be noted that isolated second reference features that are not classified into any category are also considered as a separate category.

[0083] As an example, the training method for an autoencoder is as follows: All images collected during the historical verification process are designated as historical images. A semantic segmentation neural network model is used to obtain all semantic regions in each historical image. The sub-image formed by the rectangular bounding box of each semantic region in each historical image is considered as a sample. The samples obtained corresponding to all semantic regions of all historical images are considered as a dataset.

[0084] It should be noted that the historical inspection process refers to the inspection process prior to the implementation of this embodiment. During the historical inspection process, the storage path of the video composed of all frame images was directly associated with the customs declaration data.

[0085] An autoencoder is trained using this dataset. In this embodiment, the loss function used during training is the sum of the mean squared error loss function and the feature extraction loss function. The mean squared error loss function refers to the mean squared error between the input and output of the autoencoder, which is used to ensure that the input of the autoencoder can reconstruct the input. The mean squared error loss function is well-known and will not be described in detail in this embodiment.

[0086] As an example, the method for obtaining the feature extraction loss function is: In an autoencoder, each channel of the feature map output by the encoder is considered a feature vector. The cosine similarity between each channel's feature vector and the feature vectors of all other channels is calculated, and the average is taken. This average is denoted as the similarity feature of each channel. The mean of the squares of the similarity features of all channels is used as the feature extraction loss function. The smaller the feature extraction loss function, the smaller the similarity between the channels of the feature map.

[0087] This example introduces a feature extraction loss function, which makes the channels of the feature map output by the trained autoencoder have low similarity (or even orthogonality). That is, different channels can represent different features, avoiding large repetition of features represented by different channels, and expanding the representation ability of the first reference feature for the corresponding items in the foreground region.

[0088] In this embodiment, the dataset contains 15,000 samples, the learning rate during training is set to 0.01, the optimizer is Adam, the batch size is set to 8, and the number of training iterations is set to 2,000. The specific training method is well-known and will not be described in detail in this embodiment.

[0089] It should be noted that although the semantic regions segmented by the semantic segmentation neural network model are not precise, when training an autoencoder based on samples composed of these semantic regions, since the role of the autoencoder is to extract and reconstruct features from the samples, the process of feature extraction and reconstruction will not be lost or interfered with by whether the samples can accurately represent the semantic regions of the items. Therefore, it does not affect the accuracy of the first reference feature acquisition, nor does it affect the similarity relationship between the first reference feature of the foreground region and the second reference feature of the repaired semantic region.

[0090] It should also be noted that this embodiment requires scaling each sub-image to a fixed width and height. As an example, the fixed width and height is 256×256. This embodiment uses a proportional scaling method to scale each sub-image (specifically, a linear interpolation algorithm is used for scaling), so that the maximum value of the scaled width and height of each sub-image is equal to the width or height of the fixed width and height. If the scaled sub-image cannot fill the area of ​​the fixed width and height, it is filled with pixels with a pixel value of (0, 0, 0) (filling is done on the right or bottom side of the scaled target image).

[0091] In one embodiment, the structure of the autoencoder is shown in the table below: In one embodiment, the camera that captures the goods inspection process is mounted on a shelf. It uses a fixed viewing angle, with its viewpoint pointing towards the goods inspection station. The camera's acquisition frequency is 24Hz, and each captured image frame is 1920×1080 pixels. Other camera mounting methods and camera parameters may be used in other embodiments, and this embodiment does not impose specific limitations.

[0092] In this embodiment, the camera is manually controlled to start or stop recording. For example, the inspector manually starts recording after opening the box and stops recording before sealing the box. In other embodiments, the camera is automatically controlled to start or stop recording, specifically by using a behavior detection model (e.g., the OpenPose model) to detect whether the inspector has opened or sealed the box. When the opening or sealing behavior is detected, the camera is automatically triggered to start or stop recording. The behavior detection model is existing technology and will not be described in detail in this embodiment.

[0093] It should also be noted that when using the Gaussian mixture background modeling algorithm to obtain the foreground region, since the Gaussian mixture background modeling algorithm is a soft-start process, this embodiment does not obtain the foreground region and the target semantic region within the first 50 frames of images (that is, the first 50 frames of images are not involved in the implementation of this embodiment). In some embodiments, when multiple foreground regions are detected simultaneously in each frame (for example, when an inspector picks up an item with both hands), the frame is copied multiple times, and each copied frame contains one of the multiple foreground regions, so that each frame participating in the operation of this embodiment has exactly one foreground region. Images that do not detect a foreground region (or images where the area of ​​the foreground region is less than 200) are not involved in the operation of this embodiment. However, when storing the video composed of all frames, the copied images are not stored, but the images where no foreground region was detected still need to be stored.

[0094] In this embodiment, each cargo inspection station corresponds to a camera for capturing all frame images during the inspection process. Simultaneously, each cargo inspection station corresponds to an edge device. The images captured by the camera are transmitted to the edge device via optical fiber. The edge device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program runs on the processor, it implements the steps of all the embodiments described above. Additionally, the edge device also includes a GPU for running a semantic segmentation neural network model and an autoencoder.

[0095] It should also be noted that the data tables in the database are stored on the server side, and all edge devices at the cargo inspection station are connected to the same server side via 5G technology.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for associating intelligent workstation inspection images with customs declaration data based on edge computing, characterized in that, The method includes the following steps: For all frame images collected during the cargo inspection process, the foreground region of each frame image is obtained, and the first reference feature of the foreground region of each frame image is extracted respectively. Each frame image is divided into several superpixel blocks, and each semantic region in each frame image is repaired using the superpixel blocks. The feature matching degree between the second reference feature of each repaired semantic region and all the first reference features in several preset dimensions is obtained. The difference between the repaired semantic region and the foreground region when the feature matching degree is maximized is recorded as the correction error in several dimensions. The several dimensions with the smallest correction error are recorded as the target dimension. When the second reference feature of each repaired semantic region has the highest feature matching degree with all first reference features in the target dimension, the repaired semantic region is recorded as each target semantic region in each frame image; the target semantic regions of all frames images are then associated with the customs declaration data. The methods for obtaining the aforementioned dimensions are as follows: The pixels in the foreground region or the repaired semantic region are input into the autoencoder, and the feature map output by the encoder of the autoencoder is used as the first reference feature or the second reference feature; each channel in the feature map is used as each dimension of the first reference feature or the second reference feature. The specific steps involved in using superpixel blocks to repair each semantic region in each frame of the image, and obtaining the feature matching degree between the second reference feature of each repaired semantic region and all the first reference features in several preset dimensions, are as follows: S1: The superpixel blocks through which the contour of each semantic region in each frame of an image passes are denoted as the reference block; S2: In all reference blocks of each semantic region, randomly select several neighboring superpixel blocks of each reference block to adjust the semantic region, and obtain the repaired semantic region corresponding to each semantic region; for the first reference feature obtained in any frame image, calculate the similarity between the first reference feature and the second reference feature of the repaired semantic region in several preset dimensions; the maximum value of the obtained similarity for the first reference feature of all frames image is taken as the feature matching degree. The repaired semantic region obtained when the feature matching degree is maximized refers to the repaired semantic region obtained when the feature matching degree is maximized among all the feature matching degrees obtained after repeatedly executing S2 several times. The difference between the repaired semantic region and the foreground region obtained when the feature matching degree is maximized is denoted as the correction error in several dimensions, and the specific steps included are as follows: The repaired semantic region obtained when the feature matching degree is maximized under each semantic region is recorded as the first region of each semantic region. In the first region of all semantic regions in each frame image, the first region with the largest intersection-union ratio (IU) with the foreground region of each frame image is obtained and denoted as the target inpainting region; the IU of the target inpainting region and the foreground region is denoted as the first matching coefficient of each frame image; the correction error of the several dimensions is negatively correlated with the mean of the first matching coefficient of all frame images.

2. The method for associating intelligent workstation inspection images and customs declaration data based on edge computing according to claim 1, characterized in that, The dimensions in which the correction error is minimized are denoted as the target dimensions, and the specific steps involved are as follows: Randomly select several dimensions from the first reference features and denote them as candidate dimensions, and obtain the correction error of the candidate dimensions; for all selection results of candidate dimensions, among the candidate dimensions corresponding to all selection results, denote the candidate dimension with the smallest correction error as the target dimension.

3. The method for associating intelligent workstation inspection images and customs declaration data based on edge computing according to claim 1, characterized in that, The specific steps involved in associating the target semantic regions of all frame images with customs declaration data are as follows: For all target semantic regions in all frame images, and for all target semantic regions corresponding to second reference features, all second reference features are clustered to obtain several categories. The mean of the second reference features in each category is denoted as the category center. The second reference feature with the greatest similarity to the category center in each category, and several second reference features with the least similarity to the category center are denoted as the target reference features of each category. Create a data table within the database. Each row of the data table stores data including customs declaration data, target reference features for all categories, and the storage path of the video consisting of all frame images.

4. The method for associating intelligent workstation inspection images and customs declaration data based on edge computing according to claim 1, characterized in that, The specific steps involved in randomly selecting several neighboring superpixel blocks of each reference block to adjust the semantic region, thereby obtaining the repaired semantic region for each semantic region, are as follows: For each semantic region in each frame of the image, all reference blocks are traversed. For any reference block that is traversed, several neighboring superpixel blocks are randomly expanded from the reference block to the inside or outside of the semantic region. Among them, the superpixel blocks expanded to the inside of the semantic region are called the inward pruning region, and the superpixel blocks expanded to the outside of the semantic region are called the outward pruning region. The first trimmed region is obtained by taking the union of each semantic region with the outward trimmed regions of all reference blocks. The repaired semantic region is obtained by taking the difference between the first trimmed region and the inward trimmed regions of all reference blocks.

5. The method for associating intelligent workstation inspection images and customs declaration data based on edge computing according to claim 1, characterized in that, The specific steps for calculating the similarity between the first reference feature and the second reference feature of the repaired semantic region across several preset dimensions are as follows: For any one of the preset dimensions, obtain the cosine similarity between the feature vector of the first reference feature in that dimension and the feature vector of the second reference feature in that dimension. The mean of the cosine similarities obtained for each of the preset dimensions is denoted as the similarity under the preset dimensions.

6. The method for associating intelligent workstation inspection images and customs declaration data based on edge computing according to claim 5, characterized in that, The specific steps for obtaining the feature vector are as follows: The row vector formed by the gray values ​​of each channel of the feature map serves as the feature vector of the first or second reference feature in each dimension.

7. The method for associating intelligent workstation inspection images and customs declaration data based on edge computing according to claim 6, characterized in that, The loss function used when training the autoencoder consists of a mean squared error loss function and a feature extraction loss function. The feature extraction loss function is obtained as follows: The cosine similarity between the feature vector of each channel of the feature map and the feature vectors of other channels is calculated and the average value is taken. This average value is denoted as the similarity feature of each channel. The mean of the squares of the similarity features of all channels is used as the feature extraction loss function.

Images