Semantics-based perspective image retrieval method and apparatus thereof

Abstract

The application provides a semantic-based perspective image retrieval method, a semantic-based perspective image retrieval device, an electronic device, and a computer readable medium. The method comprises: obtaining, by a perspective acquisition device, a perspective image of a space including a subject to be inspected; performing semantic division on the perspective image by a first method to obtain a plurality of semantic region units; constructing a feature extraction network by a second method; extracting, based on the perspective image and each semantic region unit of the plurality of semantic region units, a feature for each semantic region unit by the feature extraction network; and retrieving, based on the feature for each semantic region unit, an image most similar to the semantic region unit from an image feature library to assist in determining the subject to be inspected in the semantic region unit.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image recognition, and more particularly to a semantic-based perspective image retrieval method, a semantic-based perspective image retrieval device, an electronic device, and a computer-readable storage medium. BACKGROUND

[0002] It is one of the important means for customs clearance and other security businesses to use perspective images to inspect the inspected objects included in a closed space (e.g., a container, a luggage bag, a train compartment, a van, a pickup truck, etc.). The imaging principle of perspective images makes them significantly different from natural images, which increases the difficulty for an inspector to intuitively understand the images, reduces the sensitivity of the inspector to sensitive articles, and brings great challenges to the corresponding security businesses.

[0003] Therefore, there is a need for a more efficient and more accurate perspective image retrieval method. SUMMARY

[0004] To solve the problems of the prior art, the present application provides a semantic-based perspective image retrieval method and device. The method according to the present application includes three stages, namely an image semantic understanding stage, an image feature representation stage, and an image retrieval stage.

[0005] The image semantic understanding stage is to extract semantic elements from the perspective image, divide the image into visually separable independent object regions, and the independent object regions are semantic regions. The subsequent retrieval will be performed on each divided semantic region individually.

[0006] The image feature representation stage is to represent the features of each divided image semantic region. The features can describe the main information of each semantic region of the image.

[0007] The image retrieval stage is to retrieve each semantic region of the perspective image from a historical image feature library according to certain criteria.

[0008] In short, the above three stages can divide the perspective image into semantic regions, and retrieve the historical similar regions and their historical declaration information for each region to assist the inspector in comparing and analyzing each image semantic region.

[0009] The image semantic understanding stage mainly aims to split the perspective image into different semantic regions according to different visual forms or different arrangements, and all the semantic regions need to cover the entire object range of the perspective image. The related technology for solving this problem is mainly an objectness proposal method, and in the field of deep learning, the method of using a region proposal network (RPN) is more common. This stage makes the image retrieval targeted and local, and can perform fine retrieval on each independent region of the image.

[0010] The image feature representation stage mainly performs feature representation on the image semantic regions divided in the previous stage. In this stage, deep learning technology is used to represent the features of the image. Deep learning methods have greater advantages than traditional representation methods. The features extracted by traditional methods are generally fixed features designed by humans and have certain limitations. The features learned by deep learning methods based on the characteristics of the training data are more effective and adaptive, and are more beneficial to image retrieval.

[0011] The image retrieval stage takes each semantic region covering the entire image range divided in the first stage as the basic unit of retrieval. For each region, the most similar image region in the historical image is matched and the declaration information and similarity information of the similar image region are given to assist the inspection personnel in comparing each region of the perspective image, helping them to understand each region, and having the characteristics of searching each region of the mixed perspective image. It should be noted that before searching each divided region, a historical image feature library needs to be constructed in advance. Each feature in the historical image feature library takes the divided semantic region as the basic unit, and needs to have information such as description of the cargo category to which the feature belongs, region image attribution, declaration information of the image where the region is located, and object region coordinates.

[0012] Specifically, in the first aspect of the present application, a semantic-based perspective image retrieval method can include: obtaining, by a perspective acquisition device, a perspective image of a space including an inspected object; performing semantic division on the perspective image by a first method to obtain a plurality of semantic region units; constructing a feature extraction network by a second method; extracting, based on the perspective image and each semantic region unit of the plurality of semantic region units, a feature for each semantic region unit by the feature extraction network; and retrieving, based on the feature for each semantic region unit, the most similar image to the semantic region unit from an image feature library to assist in determining the inspected object in the semantic region unit.

[0013] According to the first aspect, wherein the first method is one or more of the following methods: selective search, objectness method, region proposal network (RPN).

[0014] According to the first aspect, wherein the second method is introducing a feature pyramid network (FPN) in a base network (Resnet).

[0015] According to the first aspect, wherein extracting, based on the perspective image and each semantic region unit of the perspective image, the feature for each semantic region unit by using the feature extraction network comprises: obtaining, based on the perspective image, a feature map for the perspective image by using the feature extraction network; and obtaining, based on coordinates of each semantic region unit in the perspective image, a corresponding feature for each semantic region unit in the feature map.

[0016] According to the first aspect, the method can further comprise: performing feature dimension fixing on the obtained corresponding feature for each semantic region unit, so that the corresponding feature for each semantic region unit is normalized to the same size of feature dimension.

[0017] According to the first aspect, wherein the image feature library is established based on historical perspective images without suspects, and wherein the perspective images used to establish the image feature library are recorded with one or more entry information.

[0018] According to the first aspect, wherein the image most similar to the semantic region unit is retrieved from the image feature library based on the entry information.

[0019] According to the first aspect, wherein the image most similar to the semantic region unit is a predetermined number of images.

[0020] According to the first aspect, the method can further comprise: displaying information related to the inspected object by a display unit.

[0021] In the second aspect of the present application, a semantic-based perspective image retrieval device is provided, which can comprise: a perspective acquisition module configured to obtain a perspective image of an enclosed space in which an inspected object is included; an image semantic division module configured to divide the perspective image semantically by using a first method to obtain a plurality of semantic region units; a feature extraction module configured to: construct a feature extraction network by using a second method; and extract, based on the perspective image and each semantic region unit of the perspective image, a feature for each semantic region unit by using the feature extraction network; and an image retrieval module configured to retrieve, based on the feature for each semantic region unit, an image most similar to the semantic region unit from an image feature library to help determine the inspected object in the semantic region unit.

[0022] In a third aspect of the present application, an electronic device is provided, which can include one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to the first aspect.

[0023] In a fourth aspect of the present application, a computer-readable storage medium is provided, which has stored executable instructions, which, when executed by a processor, cause the processor to implement the method according to the first aspect.

[0024] Based on the above aspects, similar images and declaration information recommendations are provided for each semantic region of the perspective image using image retrieval technology, which has a positive effect on customs clearance, luggage inspection and other security fields. The current technical means is to use pattern recognition to search for the whole region, without considering the semantic elements of the inspected object. In actual application, it is necessary to divide the semantic region as the smallest unit for searching in the case of mixed inspected objects.

[0025] The present patent considers image semantic information, extracts semantic elements from the image, searches for each semantic region of the perspective image as a basic unit, and gives similar object regions and declaration information of each semantic region, so that the inspection personnel can compare the image regions and improve the ability of auxiliary inspection. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and other embodiments and features of the present disclosure will become more apparent from the detailed description of embodiments of the present disclosure with reference to the attached drawings, in which:

[0027] Figure 1 The system architecture of the semantic-based container cargo retrieval method according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 2 The flowchart of the semantic-based container cargo retrieval method according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 3 The cargo perspective view before and after the semantic region division according to an embodiment of the present disclosure is schematically shown;

[0030] Figure 4 The flowchart of the first embodiment for extracting features of each semantic region unit according to an embodiment of the present disclosure is schematically shown;

[0031] Figure 5 The flowchart of the second embodiment for extracting features of each semantic region unit according to an embodiment of the present disclosure is schematically shown;

[0032] Figure 6 a flowchart of a third embodiment for extracting features of each semantic region unit according to an embodiment of the present disclosure is schematically shown;

[0033] Figure 7 a flowchart of a third embodiment for extracting features of each semantic region unit according to an embodiment of the present disclosure is schematically shown;

[0034] Figure 8 a block diagram of a semantic-based container cargo retrieval device according to an embodiment of the present disclosure is schematically shown; and

[0035] Figure 9 a block diagram of an electronic device adapted to implement a semantic-based container cargo retrieval method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0036] Specific embodiments of the present application will now be described in detail with reference to the following figures. Like reference numerals indicate like elements in the figures. It will be appreciated that the embodiments described below and illustrated in the drawings are only non-limiting examples of the present application. It will also be appreciated that features of the application are susceptible to variations and / or modifications other than those specifically described. The specific embodiments described are presented by way of example only and are not intended to limit the scope of the application. The drawings are not to scale and are for purposes of illustration only.

[0037] Throughout this specification, the term "one embodiment," "an embodiment," "one example," or "an example" means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the application. Therefore, the appearance of the phrases "in one embodiment," "in an embodiment," "one example," or "an example" in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0038] It will be understood that when an element is referred to as being "coupled to" or "connected to" another element, it can be directly coupled or connected to the other element or intervening elements can be present. In contrast, when an element is referred to as being "directly coupled to" or "directly connected to" another element, there are no intervening elements present.

[0039] In addition, as used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0040] It will be understood that the singular forms "a," "an," and "the" include one or more of the referenced item unless the context clearly dictates otherwise. As used herein, each of the phrases, such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C," can include all possible combinations of the items listed with those phrases. As used herein, terminology, such as "first" and "second," or "first" and "second," can be used to simply distinguish one element from another element without imposing an ordering or sequence (e.g., importance or sequence).

[0041] As used herein, the term "module" can include a unit implemented in, for example, hardware, software, or firmware, and can interchangeably be used with other terms, for example, "logic," "logic block," "part," or "circuitry." A module can be a single integral component, or a minimum unit or part thereof, adapted to perform one or more functions. For example, according to an embodiment, a module can be implemented in a form of an application-specific integrated circuit (ASIC).

[0042] It should be understood that various embodiments of the present disclosure and the terms used therein are not intended to limit technically disclosed features to specific embodiments and include various changes, equivalents, or replacements for the corresponding embodiments. Unless otherwise defined, all the terms used herein have the meanings that are similar to those of a dictionary, a textbook, and the like, commonly used at the time of filing of the present disclosure, and should be interpreted as having the meanings of the terms in the context of the related technology. Therefore, an exemplary embodiment of the present disclosure should be considered to be including various modifications, substitutions, and equivalents of the technical features disclosed herein.

[0043] In addition, it will be understood by one of ordinary skill in the art that the drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale. For the drawings, like reference numerals can be used to refer to similar or related elements. The present disclosure will now be described, by way of example, with reference to the accompanying drawings.

[0044] It should be noted that in recent years, artificial intelligence and deep learning technology have been developing continuously and gradually moving towards productization, and it has become natural to use artificial intelligence means to achieve security inspection. At present, there are technologies for using artificial intelligence means to inspect perspective images, but existing artificial intelligence means are more inclined to the intelligent direction of container inspection in perspective images, and more for risk prediction.

[0045] Image retrieval technology has been widely applied, mainly focusing on natural image retrieval and medical perspective image retrieval. At present, retrieval technology for perspective images also exists, but the existing technology uses traditional machine learning to retrieve perspective images, without considering the semantic information of perspective images, without extracting semantic elements, that is, only the whole image is retrieved, and the whole perspective image area retrieval is somewhat rough and inaccurate. However, in fact, the same kind of objects with different visual forms or multiple objects are likely to appear in the same perspective image, at which time it is necessary to retrieve each visually separable image area in the perspective image separately in order to thoroughly check each independent area.

[0046] To solve the problems as described above, embodiments of the present disclosure provide a more efficient and more accurate semantic-based perspective image retrieval method and a device for performing the method, the method comprising: obtaining, by a perspective acquisition device, a perspective image for a space including an object to be checked; performing semantic division on the perspective image by using a first method to obtain a plurality of semantic region units; constructing a feature extraction network by using a second method; extracting, based on the perspective image and each semantic region unit of the plurality of semantic region units, features for each semantic region unit by using the feature extraction network; and based on the features for each semantic region unit, retrieving, from an image feature library, an image region most similar to the semantic region unit to assist in determining or understanding the object to be checked in the semantic region unit.

[0047] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with specific embodiments.

[0048] For the sake of clarity and ease of description, in the following description, a container is taken as a specific example of a space including an object to be checked, and accordingly, goods in the container are taken as a specific example of the object to be checked. However, it should be understood by those skilled in the art that the container, the goods, and the corresponding descriptive terms (for example, a semantic-based container goods retrieval method, a semantic-based container goods retrieval device, etc.) are not limited, but are only intended to facilitate description. Those skilled in the art can replace the descriptive terms accordingly based on different use scenarios of the method.

[0049] Figure 1 The system architecture 100 of the semantic-based container goods retrieval method according to the embodiments of the present disclosure is schematically shown.

[0050] The system architecture 100 can include a perspective acquisition device 101, a container 102, a central processing unit 103, and a storage device 104, wherein the perspective acquisition device 101, the processing device 103, and the storage device 104 can communicate with each other.

[0051] The perspective view acquisition device 101 can be configured to take a perspective view of the container 102 to obtain a perspective view of the goods inside the container 102.

[0052] Although the perspective view acquisition device is shown as only one in Figure 1 one, it should be understood by those skilled in the art that the perspective view acquisition device can be two or even more. In addition, the perspective view acquisition device can be arranged at different positions relative to the container 102.

[0053] The central processing unit 103 can be configured to obtain the perspective view of the goods from the perspective view acquisition device 102, to perform semantic segmentation on the perspective view of the goods to divide the area occupied by the goods in the perspective view of the goods into a plurality of semantic area units, and to search for each semantic area unit in the plurality of semantic area units to help understand or determine the goods in each semantic area unit.

[0054] The central processing unit 103 can include a processor (not shown) and a communication module (not shown), etc.

[0055] The processor of the central processing unit 103 can be configured to perform any operation associated with the semantic-based container goods search method.

[0056] The communication module can include one or more communication processors that can operate independently of the processor (e.g., an application processor (AP)) and support wired communication and / or wireless communication. According to an embodiment of the disclosure, the communication module can include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network (e.g., a short-range communication network such as Bluetooth, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or a second network (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These various types of communication modules can be implemented as a single component (e.g., a single chip), or they can be implemented as separate components (e.g., separate chips) from each other. The wireless communication module can identify and authenticate the electronic device in a communication network, such as the first network or the second network, using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in a subscriber identification module.

[0057] In addition, the central processing unit 103 can also include a display, a microphone, etc. for displaying or announcing the relevant information (e.g., category, etc.) of the goods in the area currently being searched.

[0058] The storage device 104 can be configured to store data, instructions, etc. required and / or generated when the central processing unit 103 performs any operation associated with the semantic-based container cargo retrieval method.

[0059] In an exemplary embodiment, the storage device 104 can store historical cargo images, declaration information associated with the cargo images, image serial numbers, coordinates of each cargo region in the historical images in the images in which the cargo regions are located, features of each cargo region, cargo declaration information in each region, generalized category information characterizing the cargo in the region, commodity tariff number, etc.

[0060] The above only lists several example information stored in the storage device 104, and those skilled in the art should understand that various information can be stored in the storage device 104 according to needs, specific implementations, operations, etc.

[0061] Figure 2 A flowchart of the semantic-based container cargo retrieval method according to an embodiment of the present disclosure is schematically shown.

[0062] As shown in Figure 2 , the method includes the following operations.

[0063] In operation S201, the perspective image acquisition device performs perspective imaging on the container, thereby obtaining a cargo perspective image inside the container, so that the occupied region of the cargo can be clearly seen.

[0064] In operation S203, the central processing unit acquires the cargo perspective image from the perspective image acquisition device, and performs semantic division on the acquired cargo perspective image by using a first method, thereby dividing the image of the cargo occupied region in the cargo perspective image into a plurality of semantic region units.

[0065] In an exemplary embodiment, the plurality of semantic region units have a small degree of overlap with each other, for example, the overlap region accounts for 10%, 5%, 3% or less of each semantic region unit.

[0066] Preferably, the edges of the semantic region units overlap with each other, but there is no overlap region.

[0067] Semantic division of the cargo perspective image means that the cargo occupied region is divided into visible regions according to the visual form or arrangement of the cargo, for example, as shown in Figure 3 Semantic division of the cargo perspective image, Figure 3 each dotted box region in the figure is a semantic unit region of the cargo, and all the semantic unit regions cover the entire cargo part. The subsequent operations are all performed based on the semantic region units.

[0068] In an example embodiment, the first method for semantic region division of the cargo perspective view can include, but is not limited to, a selective search, an objectness technology, and a region proposal network (RPN) widely used in deep learning, etc.

[0069] In the present embodiment, the RPN technology will be described as an example.

[0070] The RPN technology learns a large number of manually labeled object frames to obtain a first model. During the semantic region division of the cargo perspective view, the first model is used to predict an object frame similar to the labeling, and the object region of the image is recommended.

[0071] It should be noted that due to the cargo inspection requirement, the object frame in the present application needs to cover all cargo regions, and therefore the RPN network used in the present application is different from that used in conventional target detection. The target function of the RPN network needs to be further limited based on the predicted object frame, so that the object frame finally completely covers the cargo region.

[0072] In operation S205, the central processing unit constructs a feature extraction network by using a second method. The second method is different from the first method described above.

[0073] In an example embodiment, the second method can be, but is not limited to, introducing a feature pyramid network (FPN) network structure into a basic network Resnet.

[0074] In operation S207, based on the cargo perspective view obtained in operation S201 and the plurality of semantic region units obtained in operation S203, the feature extraction network extracts the feature of each semantic region unit.

[0075] In operation S209, based on the feature of each semantic region unit, an image most similar to the semantic region unit is retrieved from an image feature library to help determine or understand the cargo in the semantic region unit.

[0076] For example, the image most similar to the semantic region unit can be one most similar image, or a plurality of images with the highest similarity.

[0077] In the present application, since the basic unit for retrieval is each semantic region unit after semantic division, the feature of each semantic region unit needs to be obtained.

[0078] Figure 4 A flowchart of a first embodiment for extracting the feature of each semantic region unit according to an embodiment of the present disclosure is schematically shown.

[0079] As shown in Figure 4 , the method comprises the following operations.

[0080] In operation S401, a normalization operation is performed on each semantic region unit, so as to obtain a semantic region unit of a fixed size.

[0081] In operation S403, the normalized semantic region unit is input into the feature extraction network.

[0082] In operation S405, the input normalized semantic region unit is processed by using the feature extraction network, so as to obtain the feature of the semantic region unit.

[0083] Figure 5 A flow chart of a second embodiment for extracting the feature of each semantic region unit according to an embodiment of the present disclosure is schematically shown.

[0084] As shown in Figure 5 , the method comprises the following operations.

[0085] In operation S501, the whole goods perspective view is input into the feature extraction network.

[0086] In operation S503, the input whole goods perspective view is processed by using the feature extraction network, so as to obtain a feature map of the whole goods perspective view.

[0087] Exemplarily, the P4 layer feature map of the whole goods perspective view is obtained by using the FPN network.

[0088] In operation S505, based on the coordinates of each semantic region unit in the whole goods perspective view, the feature of the semantic region unit is intercepted from the obtained feature map of the whole goods perspective view.

[0089] Figure 6 A flow chart of a third embodiment for extracting the feature of each semantic region unit according to an embodiment of the present disclosure is schematically shown.

[0090] As shown in Figure 6 , the method comprises the following operations.

[0091] The operations in operations S601 to S605 are consistent with the operations in operations S501 to S505 in Figure 5 , and will not be described herein again.

[0092] In operation S607, a feature dimension fixing operation is performed on the obtained feature of each semantic region unit, so as to normalize the feature of each semantic region unit into a feature dimension of the same size.

[0093] Exemplarily, the Roi-pooling can be employed to perform the feature dimension fixing operation.

[0094] Figure 7 A flowchart for performing retrieval for each semantic region unit is illustratively shown according to an embodiment of the present disclosure.

[0095] As Figure 7 shown, the method comprises the following operations.

[0096] In operation S701, an image feature library is constructed using historical images.

[0097] Exemplarily, each of the historical images used to construct the image feature library can be an image of a single category and without suspicious objects.

[0098] Exemplarily, the historical images used to construct the image feature library and the cargo perspective being retrieved can have entry information. Exemplarily, the entry information can include but is not limited to image serial number, coordinates of each semantic region unit in its cargo perspective, feature representation of each region, cargo declaration information, and generalized category information (e.g., commodity name tax number or overall feature of the image cargo) representing the cargo.

[0099] In operation S703, preselected images are retrieved from the image feature map based on the entry information of the whole cargo perspective.

[0100] In operation S705, for each semantic region unit, the image with the highest similarity is selected from the preselected images based on the feature of the semantic region unit.

[0101] In an exemplary embodiment, the image with the highest similarity can be one or more.

[0102] In an exemplary embodiment, the method of determining the image with the highest similarity can be based on the Euclidean distance.

[0103] In an alternative embodiment, all images in the feature library can be preselected images, and then the similar image(s) are found from the all images.

[0104] Figure 8 A block diagram of a semantic-based container cargo retrieval device 800 according to an embodiment of the present disclosure is illustratively shown.

[0105] As Figure 8 shown, the semantic-based container cargo retrieval device 800 can comprise a perspective acquisition module 810, an image semantic division module 830, a feature extraction module 850, and an image retrieval module 870.

[0106] The perspective view obtaining module 810 can be configured to obtain a perspective view of the goods in the container.

[0107] The image semantic division module 830 can be configured to perform semantic division on the perspective view of the goods by using a first method to obtain a plurality of semantic region units.

[0108] The feature extraction module 850 can be configured to construct a feature extraction network by using a second method, and extract features for each semantic region unit by using the feature extraction network based on the perspective view of the goods and each semantic region unit in the plurality of semantic region units.

[0109] The image retrieval module 870 can be configured to retrieve an image most similar to each semantic region unit from an image feature library based on the features for the semantic region unit to determine the category of the goods in the semantic region unit.

[0110] In addition to the above modules 810, 830, 850 and 870, the semantic-based container goods retrieval device 800 can further include other modules for performing the above-mentioned various operations correspondingly.

[0111] For example, the semantic-based container goods retrieval device 800 can further include a display module for displaying various information (e.g., showing historical similar images and matching information) to a user.

[0112] In addition, in addition to the operations performed by the modules 810, 830, 850 and 870 described above, the modules 810, 830, 850 and 870 can also adaptively perform any suitable operation among the operations described above.

[0113] For the sake of clarity and brevity, the respective modules and the corresponding operations performed therein will not be described here.

[0114] The functions of the plurality of modules according to the embodiments of the present disclosure can be implemented in one module. One module according to the embodiments of the present disclosure can be split into a plurality of modules to be implemented. The modules according to the embodiments of the present disclosure can be implemented at least in part as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable manner of hardware or firmware by integrating or packaging the circuit, or in any one of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, the modules according to the embodiments of the present disclosure can be implemented at least in part as computer program modules which can perform corresponding functions when executed.

[0115] According to an embodiment of the present disclosure, at least one of the above-described modules can be implemented at least partially in hardware, such as for example, a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System on Chip (SoC), a System in Package (SiP), a System on Processor (SoP), an Application-Specific Integrated Circuit (ASIC), or any other hardware or firmware, or any suitable combination thereof, which can be integrated or packaged in any suitable manner. Alternatively, at least one of the above-described modules can be implemented at least partially in software, hardware, and firmware, any one of which or any suitable combination thereof can be implemented in any suitable manner.

[0116] Figure 9 A block diagram of an electronic device suitable for implementing the above-described semantic-based container cargo retrieval method according to an embodiment of the present disclosure is shown schematically. Figure 9 The electronic device shown is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0117] As shown in Figure 9 The electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to programs stored in a Read-Only Memory (ROM) 902 or loaded into a Random Access Memory (RAM) 903 from a storage portion 908. The processor 901 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset and / or a dedicated microprocessor (e.g., an Application-Specific Integrated Circuit (ASIC)), etc. The processor 901 can also include an on-board memory for cache use. The processor 901 can include a single processing unit or multiple processing units for executing different actions of the method processes according to embodiments of the present disclosure.

[0118] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method processes according to embodiments of the present disclosure by executing programs in the ROM 902 and / or the RAM 903. It is noted that the programs can also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 801 can also perform various operations of the method processes according to embodiments of the present disclosure by executing programs stored in the one or more memories.

[0119] According to an embodiment of the present disclosure, the electronic device 900 can further include an input / output (I / O) interface 905 that is also connected to the bus 904. The electronic device 800 can further include one or more of the following components connected to the I / O interface 905: an input part 906 including a keyboard, a mouse, etc.; an output part 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 908 including a hard disk, etc.; and a communication part 909 including a network interface card such as a LAN card, a modem, etc. The communication part 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 910 as necessary, so that a computer program read therefrom is installed in the storage part 908 as necessary.

[0120] According to an embodiment of the present disclosure, the method flow according to the embodiments of the present disclosure can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product including a computer program carried on a computer-readable storage medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, the above-described functions defined in the system implementing the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0121] The present disclosure also provides a computer-readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, which when executed, implement the methods according to the embodiments of the present disclosure.

[0122] According to an embodiment of the disclosure, the computer readable storage medium can be a non-transitory computer readable storage medium, for example, can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In this disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the disclosure, the computer readable storage medium can include one or more memories, such as the ROM 902 and / or the RAM 903 described above, and / or one or more memories other than the ROM 902 and the RAM 903.

[0123] The flow diagrams and the block diagrams in the drawings are illustrations of possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0124] Those skilled in the art will understand that features of various embodiments and / or claims of the present disclosure can be combined and / or sub-combined, even if such combinations or sub-combinations are not expressly noted in the present disclosure. In particular, features of various embodiments and / or claims of the present disclosure can be combined and / or sub-combined without departing from the spirit and teachings of the present disclosure. All such combinations and / or sub-combinations are within the scope of the present disclosure.

[0125] The embodiments of the present disclosure are described above. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications shall fall within the scope of the present disclosure.

Claims

1. A semantic-based perspective image retrieval method, comprising: A perspective image of a space including the object being inspected is obtained by a perspective acquisition device; The perspective image is semantically segmented using the first method to obtain multiple semantic region units; The second method is used to construct a feature extraction network; Based on the perspective image and each of the plurality of semantic region units, the feature extraction network is used to extract features for each semantic region unit. as well as Based on the features of each semantic region unit, the image most similar to that semantic region unit is retrieved from the image feature library to assist in identifying the object to be inspected within that semantic region unit. Each feature in the image feature library uses a defined semantic region unit as its basic unit, and each feature in the image feature library contains information describing the cargo category, the region's image affiliation, the declaration information of the image containing the semantic region unit, and the coordinates of the object region in the image. Specifically, the feature extraction network extracts features for each semantic region unit based on the perspective image and each of the plurality of semantic region units, including: Based on the perspective image, the feature extraction network is used to obtain a feature map for the perspective image; and Based on the coordinates of each semantic region unit in the perspective image, the corresponding features for each semantic region unit are obtained in the feature map. The method further includes: The feature dimension is fixed for the corresponding features obtained for each semantic region unit, so that the corresponding features for each semantic region unit are normalized to the same size feature dimension.

2. The semantic-based perspective image retrieval method according to claim 1, wherein, The first method is one or more of the following methods: Selective search, similarity method, and Region Recommendation Network (RPN).

3. The semantic-based perspective image retrieval method according to claim 1, wherein, The second method involves introducing a Feature Pyramid Network (FPN) into the base network ResNet.

4. The semantic-based perspective image retrieval method according to claim 1, wherein, The image feature library is built based on historical perspective images without suspects, and in which, The perspective image records used to build the image feature library contain one or more entries.

5. The semantic-based perspective image retrieval method according to claim 4, wherein, Based on the entry information, the image most similar to the semantic region unit is retrieved from the image feature library.

6. The semantic-based perspective image retrieval method according to claim 1, wherein, The images most similar to the semantic region units are a predetermined number of images.

7. The semantic-based perspective image retrieval method according to claim 1 further includes: Information related to the object being inspected is displayed through the display unit.

8. A semantic-based perspective image retrieval device, comprising: The perspective acquisition module is configured to acquire a perspective image of an enclosed space that includes the object being inspected. The image semantic segmentation module is configured to perform semantic segmentation on the perspective image using a first method to obtain multiple semantic region units; The feature extraction module is configured as follows: The second method is used to construct a feature extraction network; as well as Based on the perspective image and each of the plurality of semantic region units, the feature extraction network is used to extract features for each semantic region unit. as well as The image retrieval module is configured to retrieve the image most similar to each semantic region unit from an image feature library based on features specific to that semantic region unit, in order to help identify the object to be inspected within that semantic region unit. Each feature in the image feature library uses a defined semantic region unit as its basic unit, and each feature in the image feature library contains information describing the cargo category, the region's image affiliation, the declaration information of the image containing the semantic region unit, and the coordinates of the object region in the image. The feature extraction module is further configured as follows: Based on the perspective image, the feature extraction network is used to obtain a feature map for the perspective image; Based on the coordinates of each semantic region unit in the perspective image, the corresponding features for each semantic region unit are obtained in the feature map; and The feature dimension is fixed for the corresponding features obtained for each semantic region unit, so that the corresponding features for each semantic region unit are normalized to the same size feature dimension.

9. An electronic device, comprising: One or more processors; as well as Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 7.

Images