Semantic-based image recognition system and method thereof

Abstract

The invention provides a semantic-based image recognition system, and the system can comprise a perspective image obtaining module which is configured to obtain a perspective image of a detected object in a closed environment; the semantic element extraction module is configured to extract a target semantic region for each detected object in the perspective view; a feature representation module configured to obtain a feature representation for each of the semantic regions; the decision-making module is configured to determine a target feature based on the filtering model and a predetermined threshold value so as to identify a detected object; and the display module is configured to display the information of the detected object.

Description

Technical Field

[0001] The present application relates to the field of security inspection technology, and in particular to a method for identifying objects in a closed environment, and more specifically to a semantics-based intelligent image review method for container cargo. Background Art

[0002] Container inspection is a core issue involving important aspects such as social public security and national economic interests. For example, customs clearance, tariffs, and inspections are all implemented through container inspection technology. In the current situation of increasing global trade volume, automatic container inspection technology using artificial intelligence has emerged and has gradually become a product in recent years. The main principle of automatic container inspection technology using artificial intelligence is to achieve the purpose of automatic object recognition based on information such as container perspective images and electronic customs declaration documents through deep learning-based target detection and classification.

[0003] At present, the main technologies for intelligent image review of container perspective images are customs declaration form comparison technology and entrainment detection technology. The basic principle of customs declaration form comparison technology is: use the historical images of the object to build a model for each object; for the image to be identified, extract its feature representation and match the extracted feature representation with the object model declared by its annotation information, and then determine whether the object to be identified is consistent with its annotated object based on the matching degree. The basic principle of entrainment detection technology is: based on the regular characteristics of container cargo placement, only the internal consistency of the current image is used to detect the local area with abnormalities.

[0004] However, both of these two technologies have low recognition accuracy. Based on this, a technology is needed to more accurately recognize objects in a closed environment. Summary of the invention

[0005] According to a first aspect of the present application, a semantic-based image recognition system is provided, which may include: an image acquisition module, configured to acquire an image of an object to be inspected; a semantic element extraction module, configured to extract a target semantic region for each object to be inspected in the image of the object to be inspected; a feature representation module, configured to obtain a feature representation for each semantic region in the semantic region; and a decision module, configured to determine the target feature to identify the object to be inspected based on a filtering model and a predetermined threshold.

[0006] According to a first aspect, the image acquisition module is a perspective view acquisition module for acquiring an image of an inspected object within a closed environment.

[0007] According to the first aspect, the semantic element extraction module may include: a primary semantic area division module, configured to randomly divide the prediction window for each inspected object in the perspective view; a historical semantic area model module, configured to store the historical semantic area model; and an optimization module, configured to adjust the prediction window divided by the primary semantic area division module based on the historical semantic area model.

[0008] According to the first aspect, the optimization module may also be configured to modify the prediction window based on a difference between the area of ​​the adjusted prediction window and the area of ​​the region of the detected object to obtain a target semantic region for each detected object.

[0009] According to the first aspect, the target semantic region for each detected object completely covers the detected object region with the smallest area.

[0010] According to the first aspect, the feature representation module may include: multiple semantic description modules, configured to extract multiple different region features for each target semantic region; and a region feature representation module, configured to form a region feature representation set based on multiple different region features extracted by the multiple semantic description modules.

[0011] According to the first aspect, the feature representation module may further include: a sorting module configured to sort the region feature representation set obtained via the region feature representation module.

[0012] According to the first aspect, the feature representation module may further include: a weighting module configured to assign different weight values ​​to the region feature representation set obtained by the region feature representation module.

[0013] According to the first aspect, the decision module can also be configured to perform the following operations: receive a set of regional feature representations for each regional feature from the feature representation module; based on a filtering model, filter out from the filtering model a plurality of feature entries that are most similar to the received set of regional feature representations; and determine a target feature entry from the plurality of most similar feature entries based on a predetermined threshold.

[0014] According to the first aspect, the decision module can also be configured to: calculate the matching value between the received regional feature representation set and each entry in the filtering model; select multiple feature entries with the smallest matching value as the most approximate multiple feature entries; and compare the matching value of each feature entry in the most approximate multiple feature entries with a predetermined threshold, and determine the feature entry for which the matching value is less than the predetermined threshold as the target feature entry.

[0015] In a second aspect of the present application, a semantic-based image recognition method is provided, which may include: acquiring an image of an object to be inspected; extracting a target semantic region for each object to be inspected in the image of the object to be inspected; obtaining a feature representation for each semantic region in the semantic region; and determining the target feature to identify the object to be inspected based on a filtering model and a predetermined threshold.

[0016] According to the second aspect, the image of the inspected object is a perspective image of the inspected object in a closed environment.

[0017] According to the second aspect, extracting the target semantic region for each detected object in the perspective view may include: randomly dividing a prediction window for each detected object in the perspective view; and adjusting the prediction window based on a historical semantic region model.

[0018] According to the second aspect, extracting the target semantic region for each detected object in the perspective view may further include: correcting the prediction window based on the difference between the area of ​​the adjusted prediction window and the area of ​​the region of the detected object to obtain the target semantic region for each detected object.

[0019] According to the second aspect, the target semantic region for each detected object completely covers the detected object region with the smallest area.

[0020] According to the second aspect, obtaining a feature representation for each of the semantic regions may include: extracting a plurality of different region features for each target semantic region; and forming a region feature representation set based on the plurality of different region features.

[0021] According to the second aspect, obtaining the feature representation for each of the semantic regions may further include: sorting the set of region feature representations.

[0022] According to the second aspect, obtaining the feature representation for each of the semantic regions may further include: assigning different weight values ​​to the region feature representation set.

[0023] According to the second aspect, based on a filtering model and a predetermined threshold, determining a target feature to identify an object under inspection may include: receiving a set of regional feature representations for each regional feature; based on the filtering model, filtering out from the filtering model a plurality of feature entries that are most similar to the received set of regional feature representations; and determining a target feature entry from the plurality of most similar feature entries based on a predetermined threshold.

[0024] According to the second aspect, based on the filtering model and the predetermined threshold, determining the target feature to identify the inspected object may also include: calculating the matching value between the received regional feature representation set and each entry in the filtering model; selecting multiple feature entries with the smallest matching values ​​as the most approximate multiple feature entries; and comparing the matching value for each feature entry in the most approximate multiple feature entries with the predetermined threshold, and determining the feature entry for which the matching value is less than the predetermined threshold as the target feature entry.

[0025] In a third aspect of the present application, a computer readable medium is provided, comprising program code instructions for implementing the method according to the second aspect.

[0026] This application uses the perspective image of the cargo and its customs declaration form as input, and based on the semantic division of the cargo, reconstructs the customs declaration form comparison and entrainment detection technology, and effectively combines the advantages of both, thereby improving the effectiveness of automatic judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A block diagram of an image recognition system according to an exemplary embodiment of the present application is shown.

[0028] Figure 2 A block diagram of a semantic element extraction module according to an exemplary embodiment of the present application is shown.

[0029] Figure 3 An original perspective image, a perspective image after random semantic division, a perspective image after semantic area adjustment based on historical semantic areas, and a perspective image after target semantic areas are extracted according to an exemplary embodiment of the present application are shown.

[0030] Figure 4 A block diagram of a first embodiment of a feature representation module according to an exemplary embodiment of the present application is shown.

[0031] Figure 5 A block diagram of a second embodiment of a feature representation module according to an exemplary embodiment of the present application is shown.

[0032] Figure 6 A block diagram of a third embodiment of a feature representation module according to an exemplary embodiment of the present application is shown.

[0033] Figure 7 A flowchart of a semantics-based image recognition method according to an exemplary embodiment of the present application is shown.

[0034] Figure 8 An exemplary flow chart of extracting semantic elements according to an exemplary embodiment of the present application is shown.

[0035] Fig. 9An exemplary flow chart for obtaining feature representation according to an exemplary embodiment of the present application is shown.

[0036] Fig.10 An exemplary flow chart of decision execution according to an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] The specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described herein are only for illustration and are not intended to limit the present invention. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those of ordinary skill in the art that these specific details do not have to be used to implement the present invention. In other examples, in order to avoid confusing the present invention, known circuits, materials or methods are not specifically described.

[0038] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Thus, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. Furthermore, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any appropriate combinations and / or subcombinations.

[0039] It should be understood that when an element is said to be "coupled to" or "connected to" another element, it can be directly coupled or connected to the other element or there may be intermediate elements. On the contrary, when an element is said to be "directly coupled to" or "directly connected to" another element, there are no intermediate elements.

[0040] Furthermore, as used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0041] It will be understood that a noun corresponding to a term in the singular form may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B or C" may include all possible combinations of items listed together with a corresponding one of the plurality of phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish a corresponding component from another component and do not limit the components in other respects (e.g., importance or order).

[0042] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0043] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or replacement forms for the corresponding embodiments. Unless otherwise clearly defined herein, all terms will be given their broadest possible interpretation, including the meanings implied in the specification and the meanings understood by those skilled in the art and / or defined in dictionaries, papers, etc.

[0044] In addition, it should be understood by those skilled in the art that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale. For the description of the drawings, similar reference numerals may be used to refer to similar or related elements. The present disclosure will be described exemplarily with reference to the drawings below.

[0045] In order to more accurately identify objects in a closed environment, the present application proposes an improved method, which takes the perspective image of the object to be identified and its associated annotation information (for example, the name of the item marked on the customs declaration form) as input, performs semantic division on the perspective image of the object to be identified, reconstructs the customs declaration form comparison technology and the entrainment detection technology, so as to effectively improve the recognition accuracy.

[0046] The image recognition process provided in the present application includes several stages, mainly including a perspective acquisition stage, a semantic element extraction stage, a feature representation stage, a decision stage, and a display stage. Among them, the perspective acquisition stage is a stage for obtaining a perspective view of an object in a closed environment (for example, in a container). The semantic element extraction stage is a stage for extracting (i.e., dividing) semantic regions of the acquired perspective image. The feature representation stage is a stage for obtaining feature representations of the extracted semantic regions. The decision stage is a stage for filtering the results obtained in the feature representation stage in combination with a filtering model. The display stage is a stage for notifying the user of the final recognition results.

[0047] The semantic element extraction stage is characterized in that: for the object area, the object is split into area units that can be distinguished by the naked eye according to the arrangement of the object. Compared with the prior art, the semantic element extraction stage proposed in the present application is also characterized in that: the split area units completely cover the object area with the smallest area. The technology related to the semantic element extraction stage is mainly objectness proposal (Objectness Proposal), and without loss of generality, the region proposal network (Region Proposal Net, RPN) widely used in the field of deep learning. For exemplary purposes of illustration, this application takes RPN as an example to describe the semantic element extraction process. In the present application, in order to achieve the above-mentioned "full coverage" feature, the present application further limits the objective function of the existing RPN based on the area function as a condition, which will be described in detail later.

[0048] The feature representation stage is characterized by obtaining a feature representation for the region extracted using RPN. In this application, algorithms such as a customs declaration comparison algorithm, an entrainment detection algorithm, an image statistical characteristic algorithm, and a shape characteristic extraction algorithm are used to obtain multi-faceted feature representations of the object to be detected.

[0049] The decision stage is characterized by using different filtering models to reduce the false alarm rate. In the present application, blacklist filtering and whitelist filtering are exemplarily used as filtering models. The blacklist filtering and whitelist filtering models can be sequential or performed simultaneously, and then a fusion decision is made. Specifically, a non-parametric model is selected to model the blacklist and whitelist in a similar retrieval manner.

[0050] The blacklist and the whitelist respectively include, for example, non-parametric models of the tax code (HSCODE) corresponding to each commodity. When filtering using the blacklist and the whitelist, the filtering process is defined in detail as: 1) taking "entry" as the basic unit; 2) item matching value calculation, that is, the matching value calculation between each entry and the input feature representation; 3) model matching value calculation, including the matching value calculation between the model and the input feature representation; 4) including automatic model update when adding and deleting entries. A single entry includes a semantic region feature representation and its associated original semantic information. In an embodiment, each semantic region feature representation is one-to-one corresponding to the original semantic information, that is, the feature representation is one-to-one corresponding to the historical image and its regional position. In fact, in different implementations, a one-to-one mapping can be replaced by a many-to-one mapping through clustering, parameterized models, etc. to meet real-time requirements. The non-parametric model also includes the calculation of matching values ​​between all entries and a certain input. Exemplarily, the nearest neighbor method and the Euclidean distance of the nearest neighbor are taken as examples in this application. Note that under this condition, given a nonparametric model, the input is the feature information of a region, and the output is: 1) the matching value; 2) the matching value of each entry, that is, its order.

[0051] The blacklist and whitelist functions have both online automatic learning and human-computer interaction modes to meet various practical application needs. That is, they have the functions of adding, deleting, and searching for a semantic area and its feature representation (including the association with the original information). The semantic area and its representation are reflected in the form of retrieval on the system interface. The operator can modify the blacklist and whitelist and make on-site adaptive adjustments through operations such as adding and deleting.

[0052] The automatic learning function refers to calculating and updating the blacklist model for the designated suspect area and calculating and updating the whitelist model for the non-suspect area after obtaining the accurate image judgment results of manual participation. The update includes the addition of items in the model and the update of corresponding parameters.

[0053] The human-computer interaction model refers to the semantic areas displayed on the system interface and the judgment information of each area, and retrieves and displays the corresponding best matching semantic areas in the blacklist and whitelist through the non-parametric model. You can manually select the semantic areas to be added to the model, or select the historical semantic areas to be deleted from the blacklist and whitelist. After that, the non-parametric model updates the model according to the operation content. The update includes the addition and deletion of items in the model and the update of corresponding parameters.

[0054] The matching value can be a value under different algorithms such as distance, similarity, probability, etc. In addition, it should be noted that the non-parametric model item matching value and the model matching value do not need to be the same. In the embodiment, the nearest neighbor distance is used as the most basic operation. The item matching value is the distance value between the feature representations, while the model matching value is the average value of several item distances of the nearest neighbor.

[0055] The above contents will be described in detail below with reference to the accompanying drawings. Figure 1 FIG. 1 is a block diagram of an image recognition system 10 for performing image recognition based on semantics according to an exemplary embodiment of the present application. Figure 1 As shown, the image recognition system 10 may include a perspective acquisition module 100 , a semantic element extraction module 101 , a feature representation module 102 , a decision module 103 , and a display module 104 .

[0056] As mentioned above, since the object is in a closed environment, a perspective view acquisition module 100 is required to acquire the perspective view of the object. The perspective view acquisition module 100 can adopt any device for acquiring perspective views developed in the prior art or in the future. It will not be described in detail here to avoid unnecessary confusion.

[0057] The semantic element extraction module 101 can be configured to extract the semantic region of the object in the perspective view based on the perspective view obtained by the perspective view acquisition module 100. The feature representation module 102 can be configured to extract the feature representation for each semantic region based on the semantic region extracted by the semantic element extraction module 101. The decision module 103 can be configured to use a filtering model to identify the object in each semantic region. The display module 104 can be configured to display information about the object, such as the name, whether it is consistent with the name filled in on the customs declaration form, the material, the hazard level, etc.

[0058] Figure 2 Detailed block diagram of the semantic element extraction module 101 according to an exemplary embodiment of the present application is shown. Figure 2 As shown, the semantic element extraction module 101 may include a primary semantic region division module 1010 , a historical semantic region model module 1011 , and an optimization module 1012 .

[0059] The primary semantic region segmentation module 1010 may be configured to segment the object region G in the perspective image acquired by the perspective image acquisition module 100. i (i=1,2,…,n) (for example, the area occupied by the cargo in the container) is randomly semantically divided. Exemplarily, after the primary semantic region division module 1010 performs random semantic division, a number (for example, millions, tens of millions or even more) of different random semantic regions can be obtained. Exemplarily, in this article, the random semantic region obtained by random semantic division is called "prediction window o", and the set composed of such prediction windows is "prediction window set O", O={o1,o2,o3,…,on}.

[0060] The historical semantic region model module 1011 can be configured to store the historical semantic region for each object region in the historical perspective image. The historical semantic region can be, for example, a semantic region previously obtained by artificially drawing lines on the object regions in a large number of different perspective images. When artificially obtaining the semantic region for the object, it is intended to cover the semantic region of each object with the smallest rectangular box. Of course, the present application is not limited to this, and any method can be used to obtain the historical semantic region information. In the present application, the historical semantic region is referred to as the "annotation window g", g = {g1, g2, g3, ..., gm}.

[0061] The optimization module 1012 may be configured to readjust the prediction window based on the annotation window stored in the historical semantic region model module 1011 , and further adjust the semantic region based on the difference between the area of ​​the adjusted prediction window and the area of ​​the object region.

[0062] For example, Figure 3An original perspective image, a perspective image after random semantic division, a perspective image after semantic area adjustment based on historical semantic areas, and a perspective image after target semantic areas are extracted according to an exemplary embodiment of the present application are shown.

[0063] like Figure 3 As shown in (a) in FIG. 1 , the initial perspective view acquired by the perspective view acquisition module 100 shows five object areas G1 - G5 drawn by dotted lines.

[0064] After a primary random region partitioning of the initial perspective image, one can obtain, for example, Figure 3 The perspective view shown in (b) of FIG. 1 . Reference numeral o1 denotes a prediction window for the object region G1, reference numeral o2 denotes a prediction window for the object region G2, reference numeral o3 denotes a prediction window for the object region G3, and reference numeral o4 denotes a prediction window for the object region G4. Figure 3 It can be seen from (b) that the prediction window for each of the object regions G1-G4 cannot be adapted to the object region, and during the random semantic segmentation, the prediction window for the object region G5 cannot be segmented.

[0065] Figure 3 (c) in FIG. 4 shows the prediction window readjusted according to the annotation window, and the adjusted prediction window is consistent with the annotation window. Compared with the prediction window obtained by primary random semantic partitioning shown in (b), the prediction windows o1' to o4' readjusted based on the annotation window are more suitable for the object area G1-G4. However, it can still be seen that the readjusted prediction windows o1'-o4' still cannot completely cover the object area G1-G4 with the minimum area.

[0066] In order to solve this problem, the present application further limits the objective function of the existing RPN based on the area function, thereby forming the following loss function:

[0067]

[0068] Among them, L loc Make the position difference between each prediction window and the target as small as possible; L area Make the prediction window coverage area consistent with the annotation coverage area, or understand that the predicted number is consistent with the target number.

[0069] Among them, δ(o i ,g j ) indicates the annotation window g j Is it a prediction window? i If so, then δ(o i ,g j) is 1, otherwise δ(o i ,g j ) is 0. In this application, the maximum overlap between windows can be specified. i With g j The mapping between them can be one-to-one, one-to-many, many-to-one, or many-to-many. i Corresponds to at most one g j , a g j Can correspond to multiple o i , multiple g j Can correspond to an o i , multiple g j Can correspond to multiple o i . area(·) is the area function, i.e., the coverage area. λ is the balance parameter of the two losses, which is generally set to 1. Note that formula (1) is divided into two parts. The first part uses the regression accuracy as the loss, and the second part uses the coverage of the window to the object area as the loss, so that the result can cover the cargo area to the greatest extent.

[0070] In implementation, the prediction window oi after random division is first matched with each annotation window gj stored in the database model module 1011, so that the prediction window oi is initially consistent with the annotation window gj, so that the prediction window oi' adjusted based on the annotation window is roughly similar to the object area G.

[0071] After that, the area of ​​the object region G is calculated as area(G) and the area of ​​the adjusted prediction window oi' is calculated as area(oi'). The prediction window is further modified based on the difference between area(G) and area(oi'). This modification process can be repeated until a window oi' is formed that can completely cover the object region G with the minimum area. Figure 3 As shown in (d), image semantic regions o1"-o5" for object regions G1-G5 are marked.

[0072] The above method for obtaining the semantic region of an image is merely exemplary, and those skilled in the art should be able to understand that any method or means known or to be developed in the future may be used to obtain the semantic region of an image.

[0073] Return again Figure 2 After determining the image semantic areas o1"-o5" for the object areas G1-G5, the object areas G1-G5 are associated with the image semantic areas o1"-o5" and stored in the historical semantic area model module 1011 to facilitate updating of the historical semantic areas stored in the historical semantic area model module 1011.

[0074] Figure 4A block diagram of a first embodiment of a feature representation module according to an exemplary embodiment of the present application is shown.

[0075] As shown in the figure, the feature representation module 102a may include a first semantic description module 1020 to a fifth semantic description module 1024, and a regional feature representation module 1025. Any one of the first semantic description module 1020 to the fifth semantic description module 1024 may be configured to obtain different regional features for each of the multiple semantic regions extracted by the semantic element extraction module 101.

[0076] In an exemplary embodiment, the first semantic description module 1020 may be a customs declaration comparison module using customs declaration comparison technology, which is configured to compare the semantic region extracted by the semantic element extraction module 101 with historical similar objects to obtain a similarity degree.

[0077] In an exemplary embodiment, the second semantic description module 1021 may be an entrainment detection module using an entrainment detection technique, which is configured to perform comparison between the semantic regions extracted by the semantic element extraction module 101 to obtain the degree of similarity between each region and other semantic regions.

[0078] In an exemplary embodiment, the third semantic description module 1022 may be an image characteristic statistics module for counting image characteristics, and is configured to, for example, obtain an average grayscale value or grayscale variance value of a semantic region. When the average grayscale value is obtained, it may be determined based on the average grayscale value whether the object in the region belongs to that category.

[0079] In an exemplary embodiment, the fourth semantic description module 1023 may be a shape module configured to, for example, obtain the height, width, center height, etc. of the semantic region.

[0080] In an exemplary embodiment, the fifth semantic description module 1024 may be a suspect prediction module configured to predict whether a semantic region is a suspect region based on the region features acquired by the first semantic description module 1020 to the fourth semantic description module 1023 .

[0081] In an exemplary embodiment, the region feature representation module 1025 may be configured to form a region feature representation for each object region f based on the region feature obtained by each of the first semantic description module 1020 to the fifth semantic description module 1024. i The regional feature set f i ={f ihistory ,f icurrent ,f istat ,f ishape ,f isus}.

[0082] Among them, f ihistory represents the regional features for the i-th region obtained by the first semantic description module 1020, indicating the similarity between the i-th region and historical similar goods;

[0083] f icurrent represents the regional feature for the ith region obtained by the second semantic description module 1021, indicating the degree of similarity between the ith region and other semantic regions in the perspective image;

[0084] f istat The image statistical characteristics of the i-th region may include, for example, the mean and variance of the pixels in the region. Based on the image statistical characteristics, it is possible to understand what category the material of the object in the i-th region belongs to;

[0085] f ishape Represents the shape features of the i-th region, which may include, for example, the height, width, center height, etc. of the region;

[0086] f isus Indicates whether the i-th region may be a suspect region, and the value is 0 or 1. Specifically, based on f ihistory 、f icurrent 、f istat 、f ishape It can be predicted whether the ith region is a suspect region. When the ith region is predicted to be a suspect region, f isus Set to 1, otherwise, f isus Set to 0.

[0087] although Figure 4 5 semantic description modules, namely, the first semantic description module 1020 to the fifth semantic description module 1024, are shown in FIG. However, those skilled in the art should understand that the number and type of semantic description modules included in the feature representation module are not limited thereto. Correspondingly, the regional feature set is not limited to the above exemplary regional feature set.

[0088] Figure 5 FIG. 4 is a block diagram showing a second embodiment of a feature representation module according to an exemplary embodiment of the present application. Figure 5 As shown, the feature representation module 102b may include a first semantic description module 1020 to a fifth semantic description module 1024, a region feature representation module 1025, and a sorting module 1026. Since the first semantic description module 1020 to the fifth semantic description module 1024 have been described in detail before, they will not be repeated here.

[0089] The sorting module 1026 may be configured to sort all items f in the regional feature set obtained by the regional feature representation module 1025 according to a certain rule. history 、fcurrent 、f stat 、f shape 、f sus Sort.

[0090] In one example, the f for all regional features can be divided into history Put together, the f for all regional features current Put together, the f for all regional features stat Put together, the f for all regional features shape Line up together.

[0091] In one example, f sus = 1 are placed in front, and f sus = 0 are placed at the back.

[0092] In one example, according to f stat , the regional features of the areas where the objects belong to the dangerous categories such as flammable and explosive are arranged in front, and the regional features of the areas where the objects belong to the safe categories such as non-flammable and explosive are arranged in the back.

[0093] In one example, according to f shape , the regional features of the area where the shape of the object is similar to that of a gun are ranked in front, and the other regional features are ranked behind.

[0094] In one example, the region features may be arranged only by the region number.

[0095] The above sorting methods of regional features are exemplary rather than restrictive, and those skilled in the art can actually adjust the sorting methods as needed.

[0096] Figure 6 A block diagram of a third embodiment of a feature representation module according to an exemplary embodiment of the present application is shown.

[0097] like Figure 6 As shown, the feature representation module 102c may include a first semantic description module 1020 to a fifth semantic description module 1024, a region feature representation module 1025, and a weighting module 1027. Since the first semantic description module 1020 to the fifth semantic description module 1024 have been described in detail before, they will not be repeated here.

[0098] The weighting module 1027 may be configured to weight the regional features so that a set of regional features with a large weight value is processed preferentially.

[0099] In one example, we can make f sus= 1, the weight value of the feature set is large, so that f sus = 0 has a smaller weight value.

[0100] In one example, the f shape The weight values ​​of the feature set indicating that the object in the area is similar to a gun are set to be large, so that f shape The weight value of the feature set indicating that the objects in the area are similar to stone is set to be small.

[0101] In one example, the f stat The weight value of the feature set indicating that the pixel grayscale value of the object in the area is closer to that of oil is set to a larger value, and the f stat The weight value of the feature set indicating that the pixel grayscale value of the object in the area is closer to that of wood is set to be smaller.

[0102] The above weighting methods are exemplary rather than restrictive, and those skilled in the art can adjust the sorting method as needed.

[0103] Return again Figure 1 The decision module 103 may be configured to filter the regional feature set using a non-parametric filtering model, and compare the nearest neighbor matching value in the filtered feature set with a predetermined threshold to determine whether the region is a suspect region.

[0104] The filtering model can adopt a blacklist model or a whitelist model. Assume that the blacklist model is B and the whitelist model is W. In the exemplary embodiment, the input semantic information is the commodity tax number or HSCODE, indicating the specific category of the commodity. Therefore, assuming that there are N categories of commodities, B and W are respectively i (i=1,2,…,N) and w i (i=1,2,…,N). b i With w i The configurations are exactly the same, differing only in semantics and algorithmic details. i Take an example to illustrate the model composition.

[0105] Model entries and corresponding semantic descriptions

[0106] Model i The entries in are the features f of all semantic regions of this category. i , and f i The corresponding semantic description d i .d i The source can be customs declaration data, and d iThe necessary content can be a unique index for the perspective image, and through this index, the original perspective image can be uniquely found. In addition, the packaging form, import or export, manufacturer, etc. can also be recorded as semantic information in d i to achieve a more complex semantic description.

[0107] Note that for simplicity, in this exemplary embodiment, f i and d i are described as a one-to-one correspondence. In fact, f i and d i can be a one-to-many mapping, many-to-many mapping, or many-to-one mapping. For example, f i can be the mean value after a certain operation, while d i information does not have operation characteristics such as the mean value. Those skilled in the art are well aware of such implementation changes and will not be elaborated here.

[0108] The number N of the above models is approximately in the order of ten thousand, and the number of entries for each model w i is approximately in the order of one hundred thousand.

[0109] Exemplarily, the principle of the decision module 103 is illustrated below using the whitelist model as an example.

[0110] The whitelist model is a set W composed of w i (i = 1, 2,..., N), where W as described above contains several entries f i and their d i (i = 1, 2,..., N). Assume that the input region feature is f x , where f x can be any region feature obtained through the feature representation module 102.

[0111] The decision module 103 can be configured to calculate the matching value s x between the input region feature f i and each f i in the whitelist model W, where s i =(f x -f i ) 2 . Based on such a calculation, N matching values can be obtained.

[0112] The decision module 103 can also be configured to sort the calculated matching values and select M (M < N) (for example, M = 10) matching values from these matching values as the most approximate matching values.

[0113] The decision module 103 may also be configured to form a matching set based on the selected M most approximate matching values, and the regional feature f and its semantic description d associated with each most approximate matching value. xi}, p xi =[f top , d top ,s top ] i .

[0114] In addition, the decision module 103 can also be configured to match each s in the set P top With the predetermined threshold T W is compared, and the value less than the predetermined threshold value T W s top The target match value is selected and the f associated with the target match value is top Determined as the target region feature, d is associated with the target matching value top Determine the target semantic description.

[0115] The above is a description of the whitelist model. The processing of the blacklist model is similar to that of the whitelist model, except for the following differences:

[0116] 1) The regional features input into the model are the areas considered suspicious;

[0117] 2) The number of entries in the blacklist model is much smaller than the number of entries in the whitelist model;

[0118] 3) and whitelist threshold T W Similarly, the predetermined threshold of the blacklist is T B When the closest matching value in the matching set is less than the predetermined threshold T B , it is determined to be the target matching value.

[0119] In addition, in an exemplary embodiment, after the decision module 103 determines the target set, the decision module 103 can update the filtering model based on the determination result between the input and the output. It is known to those skilled in the art that the update can be performed through online automatic update, manual update of human-computer interaction, or a combination of the two.

[0120] Return again Figure 1 The display module 104 may be configured to display the target matching set determined by the decision module 103. The display module 104 may also display the current image to be tested, the image semantic region, the most matching historical image and its corresponding region, and the matching value, but is not limited thereto.

[0121] Based on the above description, a semantics-based image recognition method will be generated in detail. Figure 7A flowchart of a semantics-based image recognition method according to an exemplary embodiment of the present application is shown.

[0122] At step 71 , the perspective view acquisition module 100 acquires a perspective view of an object in a closed space.

[0123] At step 72 , the semantic element extraction module 101 may obtain a semantic region about each object in the image based on the perspective view, wherein the goal of the semantic region is to completely cover the object with a minimum area.

[0124] At step 73 , the feature representation module 102 may extract a region feature for each semantic region extracted by the semantic element extraction module 101 .

[0125] At step 74 , the decision module 103 may filter out the most matching features for each regional feature based on the filtering model, and determine which of the filtered features is the target feature based on a comparison with a predetermined threshold.

[0126] At step 705 , any information associated with the target feature decided by the decision module 103 is displayed.

[0127] At this point, the semantic-based image recognition process is completed.

[0128] Figure 8 An exemplary flow chart of extracting semantic elements according to an exemplary embodiment of the present application is shown.

[0129] At step 81 , the primary semantic region segmentation module 1010 forms a prediction window for each object in the perspective image.

[0130] At step 82, the optimization module 1012 readjusts the prediction window based on the annotation window stored in the historical semantic region model module 1011. Thereafter, at step 83, the optimization module 1012 further adjusts the window based on the difference between the area of ​​the adjusted prediction window and the area of ​​the object region.

[0131] At step 84, the optimization module 1012 determines whether the area of ​​the prediction window is the same as the area of ​​the object region. If they are equal ("yes" at step 84), the window can completely cover the object region with the minimum area, thereby obtaining the target semantic region (step 85). If they are not equal ("no" at step 84), the operation at step 83 is performed again. This process is repeated until the area of ​​the adjusted prediction window is the same as the area of ​​the object region.

[0132] When the target semantic region is obtained (step 85), the optimization module 102 updates the obtained target semantic region into the historical semantic region model so as to be used for adjusting other prediction windows later.

[0133] Fig. 9 An exemplary flow chart for obtaining feature representation according to an exemplary embodiment of the present application is shown.

[0134] In step 91, for each semantic region, a plurality of different regional features are obtained using a plurality of different technologies. In an exemplary embodiment, the obtained plurality of different regional features may include regional features obtained by a customs declaration form comparison algorithm, regional features obtained by an entrainment detection algorithm, image statistical characteristics, shape features, and suspicion prejudgment, but the regional features are not limited thereto and may include more other regional features.

[0135] At step 92 , the feature representation module 1025 forms a feature representation set for each semantic region based on the multiple different region features obtained at step 91 .

[0136] Optionally, at step 93, the feature representation set formed for each of the multiple semantic regions at step 92 may be weighted or sorted, so that a feature representation set with a higher sorting or a larger weight value is processed preferentially.

[0137] Fig.10 An exemplary flow chart of decision execution according to an exemplary embodiment of the present application is shown.

[0138] In step 101, a filtering model is used to filter out several feature representations that are most similar to the input feature to be identified. In an exemplary embodiment, several most similar feature representations can be determined, for example, by calculating a matching value between the feature representation in the filtering model and the input feature representation to be identified.

[0139] At step 102, a target feature representation is selected from a plurality of closest feature representations based on a predetermined threshold. In an exemplary embodiment, a feature representation smaller than a predetermined threshold among the plurality of closest feature representations may be determined as a target feature representation. In this case, the target feature representation is the identified feature representation.

[0140] In step 103, information associated with the target feature representation is displayed, which may include, for example, the current image to be tested, the image semantic region, the matching most similar historical image, the corresponding region, the matching value, etc.

[0141] Although the various steps are described above according to the order shown in the drawings, those skilled in the art should understand that the various steps may be performed in a different order, or the embodiments of the present invention may be implemented without one or more of the above steps.

[0142] It can be understood from the foregoing that the electronic components of one or more systems or devices may include, but are not limited to, at least one processing unit, a memory, and a communication bus or communication device that couples the various components including the memory to the processing unit. The system or device may include or may access various device-readable media. The system memory may include device-readable storage media in the form of volatile and / or non-volatile memory (e.g., read-only memory (ROM) and / or random access memory (RAM)). By way of example and not limitation, the system memory may also include an operating system, application programs, other program modules, and program data.

[0143] Embodiments may be implemented as systems, methods, or program products. Thus, embodiments may take the form of all-hardware embodiments or embodiments that include software (including firmware, resident software, microcode, etc.), which may be collectively referred to herein as "circuits," "modules," or "systems." In addition, embodiments may take the form of a program product embodied in at least one device-readable medium having device-readable program code embodied thereon.

[0144] Combinations of device-readable storage media may be used. In the context of this document, a device-readable storage medium ("storage medium") may be any tangible, non-signal medium that may contain or store a program consisting of program code configured to be used by or in conjunction with an instruction execution system, apparatus, or device. For purposes of this disclosure, a storage medium or device should be interpreted as being non-transitory, i.e., not including signal or propagation media.

[0145] The present disclosure is presented for purposes of illustration and description, but is not intended to be exhaustive or limiting. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to illustrate the principles and practical applications, and to enable those of ordinary skill in the art to understand the various embodiments of the present disclosure with various modifications suitable for the intended specific use.

Claims

1. A semantic-based image recognition system, comprising: An image acquisition module configured to acquire a perspective view of an object under inspection in a closed environment; A semantic element extraction module is configured to extract a target semantic region of each inspected object in the perspective view of the inspected object; a feature representation module, configured to obtain a feature representation for each of the target semantic regions; as well as The decision module is configured as follows: receiving a set of region feature representations for each target semantic region from the feature representation module; Calculating a matching value between the received regional feature representation set and each entry in the filtering model, and selecting a plurality of feature entries with the smallest matching value as the most approximate plurality of feature entries; as well as The matching value for each of the most similar feature entries is compared with the predetermined threshold, and the feature entry for which the matching value is less than the predetermined threshold is determined as a target feature entry to identify the detected object. Based on the filtering model and a predetermined threshold, target features are determined to identify the inspected object.

2. The image recognition system according to claim 1, wherein: The semantic element extraction module comprises: A primary semantic region segmentation module, configured to randomly segment a prediction window for each detected object in the perspective image; A historical semantic region model module, configured to store the historical semantic region model; and The optimization module is configured to adjust the prediction window divided by the primary semantic region division module based on the historical semantic region model.

3. The image recognition system according to claim 2, wherein: The optimization module is further configured to modify the prediction window based on a difference between an area of ​​the adjusted prediction window and an area of ​​the region of the detected object, so as to obtain the target semantic region for each detected object.

4. The image recognition system according to claim 3, wherein: The target semantic region for each detected object completely covers the detected object region with the smallest area.

5. The image recognition system according to claim 1, wherein: The feature representation module comprises: A plurality of semantic description modules, configured to extract a plurality of different region features for each target semantic region; and The region feature representation module is configured to form a region feature representation set based on the multiple different region features extracted by the multiple semantic description modules.

6. The image recognition system according to claim 5, wherein: The feature representation module further includes: a sorting module configured to sort the region feature representation set obtained by the region feature representation module.

7. The image recognition system according to claim 3, wherein: The feature representation module further includes: a weighting module configured to assign different weight values ​​to the region feature representation set obtained by the region feature representation module.

8. A semantic-based image recognition method, the method comprising: Obtaining a perspective view of the inspected object within the closed environment; Extracting a target semantic region of each inspected object in the perspective view of the inspected object; Obtaining a feature representation for each target semantic region in the target semantic region to form a region feature representation set; receiving a set of region feature representations for each target semantic region from the feature representation module; Calculating a matching value between the received regional feature representation set and each entry in the filtering model, and selecting a plurality of feature entries with the smallest matching value as the most approximate plurality of feature entries; as well as The matching value for each of the most similar feature entries is compared with the predetermined threshold, and the feature entry for which the matching value is less than the predetermined threshold is determined as a target feature entry to identify the detected object.

9. The image recognition method according to claim 8, wherein: The extracting of the target semantic region for each detected object in the perspective image comprises: randomly dividing a prediction window for each detected object in the perspective image; and The prediction window is adjusted based on a historical semantic region model.

10. The image recognition method according to claim 9, wherein: The extracting of the target semantic region for each detected object in the perspective view further comprises: The prediction window is modified based on the difference between the area of ​​the adjusted prediction window and the area of ​​the region of the detected object to obtain the target semantic region for each detected object.

11. The image recognition method according to claim 10, wherein: The target semantic region for each detected object completely covers the detected object region with the smallest area.

12. The image recognition method according to claim 8, wherein: The obtaining of a feature representation for each of the semantic regions comprises: Extracting a plurality of different region features for each target semantic region; and Based on the multiple different regional features, a regional feature representation set is formed.

13. The image recognition method according to claim 12, wherein: The obtaining of the feature representation for each of the semantic regions further includes: sorting the region feature representation set.

14. The image recognition method according to claim 12, wherein: The obtaining of the feature representation for each of the semantic regions further includes: assigning different weight values ​​to the region feature representation set.

15. A computer readable medium comprising program code instructions for implementing the method according to any one of claims 8 to 14.

Images