Museum exhibit identification method and device, electronic equipment and storage medium

Abstract

The invention discloses a museum exhibit identification method and device, electronic equipment and a storage medium. The method comprises the steps that a target image is acquired, and the target image comprises an exhibit to be recognized; single-subject detection is carried out on the target image, a subject area image in the target image is obtained, and the subject area image is the part, containing the to-be-recognized exhibit, in the target image; performing feature extraction on the target image to obtain global features, and performing feature extraction on the main region image to obtain local features; according to the global features and the local features, matching retrieval is carried out in an exhibit feature library to obtain exhibit information corresponding to the target image, the exhibit feature library comprises feature vectors of a plurality of exhibits, and the feature vectors are determined according to the global features and the local features of the exhibits. The museum exhibit identification method solves the technical problem of poor identification precision of a museum exhibit identification method in related technologies.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and more specifically, to a method, device, electronic device, and storage medium for identifying museum exhibits. Background Art

[0002] With the rapid development of digitalization and intelligentization, museums, as important platforms for displaying culture and educating the public, are also gradually moving towards intelligentization. Museum digitization projects mainly include the digitization of collections, exhibitions, and services. Among them, exhibit identification is a key component of exhibition digitization.

[0003] However, most of the museum exhibit recognition methods in related technologies have poor recognition effects and have technical problems such as low recognition accuracy and poor system versatility.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present application provide a method, device, electronic device, and storage medium for identifying museum exhibits, so as to at least solve the technical problem of poor recognition accuracy of museum exhibit recognition methods in related technologies.

[0006] According to one aspect of an embodiment of the present application, a method for identifying museum exhibits is provided, comprising: acquiring a target image, wherein the target image contains an exhibit to be identified; performing single-subject detection on the target image to obtain a subject area image in the target image, wherein the subject area image is the portion of the target image containing the exhibit to be identified; performing feature extraction on the target image to obtain global features, and performing feature extraction on the subject area image to obtain local features; performing matching retrieval in an exhibit feature library based on the global features and the local features to obtain exhibit information corresponding to the target image, wherein the exhibit feature library contains feature vectors of multiple exhibits, and the feature vectors are determined based on the global features and the local features of the exhibits.

[0007] Optionally, performing single-subject detection on the target image to obtain a subject area image in the target image includes: using a single-subject detection model to analyze the target image to obtain a detection frame and a confidence level corresponding to the detection frame, wherein the detection frame is used to select the exhibit as the subject in the target image; cropping the target image according to the coordinate information of the detection frame with the highest confidence level to obtain a subject area image; and when the contrast of the subject area image is lower than a preset contrast threshold, performing histogram equalization processing on the subject area image to enhance the contrast of the subject area image, wherein the contrast of the enhanced subject area image is not lower than the preset contrast threshold.

[0008] Optionally, matching in the exhibit feature library based on global features and local features includes: determining a similarity parameter between a feature vector of each exhibit in the exhibit feature library and a target feature corresponding to the target image, wherein the target feature includes: a global feature and a local feature, or a fusion feature obtained by fusing the global feature and the local feature, and the similarity parameter is used to characterize the degree of similarity between the feature vector and the target feature; determining a maximum preset number of exhibits in the exhibit feature library with similarity parameters to the target feature as candidate exhibits; if the similarity parameter of the candidate exhibit is greater than a preset similarity threshold, determining that the target image and the candidate exhibit are successfully matched; if the similarity parameter of the candidate exhibit is greater than the preset similarity threshold, determining that the target image has failed to match, and returning a first prompt message to the terminal device that uploaded the target image, wherein the first prompt message is used to indicate that exhibit information of the exhibit was not found.

[0009] Optionally, after determining that the target image and the candidate exhibit are successfully matched, the method further includes: determining a similarity parameter between a target feature of the target image and a target feature of each historical misjudgment image in a historical misjudgment list; if there is a historical misjudgment image in the historical misjudgment list whose second similarity parameter is greater than the first similarity parameter, returning a first prompt message to the terminal device that uploaded the target image, wherein the second similarity parameter is a similarity parameter between the target feature of the target image and the target feature of the historical misjudgment image, and the first similarity parameter is a similarity parameter between the target feature of the target image and a feature vector of the candidate exhibit; if there is no historical misjudgment image in the historical misjudgment list whose second similarity parameter is greater than the first similarity parameter, obtaining exhibit information of the candidate exhibit and sending the exhibit information to the terminal device, wherein the exhibit information includes: an explanation word corresponding to the candidate exhibit.

[0010] Optionally, after sending the exhibit information to the terminal device, the method also includes: upon receiving a first feedback message sent by the terminal device, determining the target image as a historical misjudgment image, and storing the historical misjudgment image and the target features of the historical misjudgment image in a historical misjudgment list, wherein the first feedback message is used to characterize that the exhibit information does not match the exhibit contained in the target image.

[0011] Optionally, the method also includes: determining an exhibit level corresponding to the exhibit in the target image, wherein the exhibit level includes: museum level, exhibition level, explanation point level, and cultural relic level; obtaining exhibit data corresponding to the exhibit level in the exhibit feature library; and matching the exhibit data corresponding to the exhibit level based on global features and local features to obtain exhibit information corresponding to the target image.

[0012] Optionally, the exhibit feature library further includes: an exhibition location and an exhibition time of the exhibit; the method further includes: determining the shooting location and the shooting time of the terminal device that uploaded the target image when shooting the target image; obtaining candidate exhibit data in the exhibit feature library whose exhibition location is within a target area and whose exhibition time is within a target time range, wherein the target area is an area with a preset length as the radius centered on the shooting location, and the target time range is a time range within a preset time length before and after the shooting time; matching is performed in the candidate exhibit data based on global features and local features to obtain exhibit information corresponding to the target image.

[0013] According to another aspect of an embodiment of the present application, a museum exhibit identification device is also provided, including: an image acquisition module, used to acquire a target image, wherein the target image contains an exhibit to be identified; a subject detection module, used to perform single subject detection on the target image to obtain a subject area image in the target image, wherein the subject area image is the portion of the target image containing the exhibit to be identified; a feature extraction module, used to perform feature extraction on the target image to obtain global features, and perform feature extraction on the subject area image to obtain local features; an exhibit matching module, used to perform matching retrieval in an exhibit feature library based on global features and local features to obtain exhibit information corresponding to the target image, wherein the exhibit feature library contains feature vectors of multiple exhibits, and the feature vectors are determined based on the global features and local features of the exhibits.

[0014] According to another aspect of the embodiments of the present application, an electronic device is provided, including: a memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program executes a method for identifying museum exhibits when the program is run.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the museum exhibit identification method by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps of the museum exhibit identification method when executed by a processor.

[0017] In an embodiment of the present application, a target image is acquired, wherein the target image contains an exhibit to be identified; single subject detection is performed on the target image to obtain a subject area image in the target image, wherein the subject area image is the portion of the target image containing the exhibit to be identified; feature extraction is performed on the target image to obtain global features, and feature extraction is performed on the subject area image to obtain local features; matching retrieval is performed in an exhibit feature library based on the global features and the local features to obtain exhibit information corresponding to the target image, wherein the exhibit feature library contains feature vectors of multiple exhibits, and the feature vectors are determined based on the global features and the local features of the exhibits. By combining the global features and the local features for matching, the purpose of improving the recognition accuracy is achieved, thereby solving the technical problem of poor recognition accuracy of the museum exhibit recognition method in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for identifying museum exhibits according to an embodiment of the present application;

[0020] Figure 2 1 is a schematic diagram of a method flow for identifying museum exhibits according to an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of an application scenario and overall architecture of a method for intelligent identification of museum exhibits based on multi-feature fusion and dynamic constraints provided in accordance with an embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of an image processing flow on a server side provided according to an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of dual-modal feature extraction and fusion provided according to an embodiment of the present application;

[0024] Figure 6 This is a schematic diagram of a voting decision process provided according to an embodiment of the present application;

[0025] Figure 7 This is a schematic diagram of a closed-loop blacklist mechanism principle provided according to an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of a hierarchical exhibit classification system provided according to an embodiment of the present application;

[0027] Figure 9 Schematic diagram of the structure of a museum exhibit identification device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] To facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns involved in the embodiments of the present application are explained as follows:

[0031] Single Mainbody Detection: A technique for locating the most prominent object region in an input image, with the output being the coordinates of a detection box.

[0032] Open-Set Retrieval: Performs feature matching in scenarios where the candidate set includes unknown categories (unstocked exhibits). In addition to distinguishing known categories, it also needs to distinguish between known and unknown categories.

[0033] Closed-Set Retrieval: The candidate set only contains a limited number of pre-defined categories, and the model only needs to select the best match among the known categories.

[0034] Hierarchical exhibit classification system: Defines the four levels and affiliation of exhibits in museum scenarios: "museum, exhibition, explanation point, cultural relic", where an explanation point can contain one or more cultural relics (a one-to-many relationship).

[0035] "Layout" exhibits: Different arrangements of similar small exhibits create different categories of exhibits. These are difficult to distinguish using cropped image features of individual small exhibits, requiring global image features. For example, identical lanterns arranged in different layouts, hung in temporary corridors or on wooden buildings, create lantern displays with distinct themes.

[0036] Independent Cultural Relics: "Cultural Relics" are the lowest level of exhibits in the hierarchical exhibit classification system and cannot be further subdivided. Independent cultural relics have specific properties, meaning that independent cultural relics of the same style cannot be placed in different categories.

[0037] HDR (High Dynamic Range) is a technology that improves the dynamic range of an image by combining multiple photos with different exposures. It is mainly used to solve the problem of significant contrast between bright and dark areas in the scene.

[0038] In related technologies, methods for identifying museum exhibits mainly include:

[0039] 1) QR code recognition method: QR codes are posted near exhibits, and visitors can identify the corresponding cultural relics by scanning the QR codes. This method has poor flexibility and low efficiency. In scenes with a large number of tourists, it is easy to get crowded and difficult to scan the codes.

[0040] 2) Global feature matching: Use a pre-trained model to extract global features from the entire image and retrieve the top-1 matching items in the vector library using cosine similarity. This method is susceptible to background interference during recognition, such as crowds visiting an ancient museum.

[0041] 3) Multi-target detection + local matching: Deploy a detection model to detect multiple candidate areas and perform feature matching independently for each area. This method requires continuous updating of the detection model to adapt to new exhibit shapes, significantly increasing deployment costs. In addition, for "layout" exhibits, this recognition method may lose global features, increasing the degree of confusion in resolution.

[0042] From the above, it can be seen that it is difficult to strike a balance between recognition accuracy and system versatility in the recognition solutions in the related art. To cover most exhibit categories while ensuring accuracy, the model needs to be updated frequently. In addition, the recognition solutions in the related art face a dilemma in image area selection: in the whole image extraction mode, when the exhibits occupy a relatively small area in the picture (such as long-distance shooting), the feature model is difficult to focus on the effective area, and the recognition accuracy drops significantly; while in the local cropping mode: when cropping single exhibits for "layout-type" exhibits (such as the placement of lanterns), the spatial relationship features between exhibits will be lost. In addition, for open domain retrieval, when the exhibit corresponding to the query image is not in the library, if the vector library contains relatively similar different exhibits, the solutions in the related art are prone to mistakenly match the similar different exhibits instead of matching to empty (i.e., unknown category).

[0043] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0044] According to an embodiment of the present application, an embodiment of a method for identifying museum exhibits is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or electronic device) for implementing a method for identifying museum exhibits is shown. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0046] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0047] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the museum exhibit identification method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned museum exhibit identification method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0048] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0049] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0050] In the above operating environment, the embodiment of the present application provides a method for identifying museum exhibits. Figure 2 FIG. 1 is a schematic diagram of a method flow for identifying museum exhibits according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0051] Step S202: acquiring a target image, wherein the target image includes an exhibit to be identified;

[0052] Step S204: performing single subject detection on the target image to obtain a subject area image in the target image, wherein the subject area image is the portion of the target image containing the exhibit to be identified;

[0053] Step S206, performing feature extraction on the target image to obtain global features, and performing feature extraction on the subject area image to obtain local features;

[0054] Step S208 , performing a matching search in an exhibit feature library based on the global features and local features to obtain exhibit information corresponding to the target image, wherein the exhibit feature library contains feature vectors of multiple exhibits, and the feature vectors are determined based on the global features and local features of the exhibits.

[0055] Through the above steps, by combining global features and local features for matching, the purpose of improving recognition accuracy is achieved, thereby solving the technical problem of poor recognition accuracy of museum exhibit recognition methods in related technologies.

[0056] The following further introduces the museum exhibit identification method in steps S202 to S208 of the embodiment of the present application.

[0057] Figure 3 This is a schematic diagram of an application scenario and overall architecture of a method for intelligent identification of museum exhibits based on multi-feature fusion and dynamic constraints according to an embodiment of the present application, such as Figure 3 As shown in the figure, the architecture includes: a mobile-side interaction module, a server-side recognition engine, and a database. The mobile-side interaction module can be deployed in a mobile app to implement functions such as taking photos, image compression (JPEG compression), and displaying exhibit information returned by the server (for example, exhibit explanations). The server-side recognition engine includes: a single-subject detection model, a feature extraction model, a voting decision module, and a blacklist management function.

[0058] The database contains a feature vector database (i.e., an exhibit feature database) and an explanation word database. The feature vector database is used to store the multi-angle sample feature vectors (global + local dual vectors) of the hierarchical exhibit classification system, and can also be accompanied by spatiotemporal metadata (such as the classification system level and specific category of the exhibit, exhibition time period, exhibition hall location, etc.). In addition, the exhibit features of the blacklist (i.e., the historical misjudgment list) can also be stored here and marked with a blacklist; in the explanation word database, the corresponding explanation word can be indexed based on the identified exhibit classification level and specific category.

[0059] With the help of Figure 3As shown in the architecture, when a visitor uses the system to take a photo of a scene or exhibit at any level in the hierarchical exhibit classification system, the system can identify the dominant exhibit in the picture and return the information of the associated explanation points; when adding exhibits to a temporary exhibition, the administrator only needs to upload the new sample features to the vector database to complete the system upgrade, and when removing an outdated exhibit, the administrator only needs to remove the corresponding features from the vector database to complete the system update.

[0060] The following is a further introduction to the server's processing flow after obtaining the target image uploaded by the tourist on the mobile terminal. Figure 4 As shown, after taking a photo, visitors complete image quality compression in the mini-program and upload it to the server. The server then performs single-subject detection, extracting local and global features from the cropped subject area image and the entire image using the same feature extraction model. The server then searches the vector database (exhibit feature library) for matching scores with each vector, using voting to perform robust matching. If a match is successful, a blacklist filter check is performed. Finally, the corresponding explanation for the exhibit is returned.

[0061] The specific steps of extracting and fusing bimodal features (i.e. local features and global features) are as follows: Figure 5 As shown, further introduction is given below.

[0062] First, the steps of local feature extraction are as follows.

[0063] In some embodiments of the present application, single-subject detection is performed on a target image to obtain a subject area image in the target image, including: using a single-subject detection model to analyze the target image to obtain a detection frame and a confidence level corresponding to the detection frame, wherein the detection frame is used to select the exhibit as the subject in the target image; cropping the target image according to the coordinate information of the detection frame with the highest confidence level to obtain a subject area image; and when the contrast of the subject area image is lower than a preset contrast threshold, performing histogram equalization processing on the subject area image to enhance the contrast of the subject area image, wherein the contrast of the enhanced subject area image is not lower than the preset contrast threshold.

[0064] Specifically, a single-subject detection model of any structure can be used to output detection frame coordinates and confidence scores. The target image is then cropped based on the detection frame coordinates to obtain a subject area image. As an optional implementation, this subject area image can be enhanced. Specifically, because mobile devices may not have HDR, images captured within glass display cases may exhibit low contrast. Therefore, the cropped image can be subjected to image enhancement operations, including but not limited to histogram equalization.

[0065] Afterwards, based on the single-subject detection results, a unified metric learning model (cross-domain pre-training optimization) can be used to extract the global features of the entire image (capturing the spatial layout of the exhibits and fixed background clues) and the local features of the cropped area (focusing on detailed textures) in parallel, and construct a multi-granularity complementary feature space, that is, to perform dual-path feature extraction on the enhanced subject area image and the entire target image.

[0066] In this embodiment, a feature extraction model pre-trained using metric learning on a variety of open source datasets can be used to extract features from two granularity images, obtaining local features and global features, respectively. For example, for the whole image branch, the original target image can be resized to 224*224, input into the feature extraction network, and 512-dimensional global features can be extracted (capturing fixed display case background clues and exhibit space layout); for the cropping branch, the main area image can be resized to 224*224, input into the feature extraction network, and 512-dimensional local detail features can be extracted (focusing on microstructures such as bronze ornamentation).

[0067] After obtaining the local features and the overall features, you can choose to fuse the two features into one feature; or you can maintain these two separate features in the system; this is not limited in the present embodiment. If feature fusion is used, weighted fusion or direct splicing can be used, where the weighted fusion method is shown in the following formula:

[0068] F_fused=W_g⊙F_global+(1-W_g)⊙F_local

[0069] Here, F_global and F_local represent the features extracted by the feature extraction model for the entire image (global features) and the features extracted by the feature extraction model for the cropped image (local features), respectively. F_fused is the fused feature, and W_g is the fusion weight, which can be dynamically determined based on actual needs.

[0070] Based on the results of single-subject detection, the present embodiment employs the same metric learning model to concurrently extract global features (capturing the spatial layout and background cues of the exhibits) and cropped local features (focusing on microtextures). This approach, including but not limited to weighted fusion or independent retention of both features, constructs a complementary feature space. This approach resolves the discrepancy between identifying "layout-based exhibits" or "assemblies of cultural relics" and "independent cultural relics," comprehensively reducing misidentification issues in these scenarios. Furthermore, it can enhance the distinction between similar cultural relics by using a fixed display case background (such as the reflective characteristics of a glass display case), significantly improving recognition accuracy.

[0071] Afterwards, matching retrieval is performed in the exhibit feature library based on global features and local features, as follows.

[0072] In some embodiments of the present application, matching in an exhibit feature library based on global features and local features includes: determining a similarity parameter between a feature vector of each exhibit in the exhibit feature library and a target feature corresponding to a target image, wherein the target feature includes: a global feature and a local feature, or a fused feature obtained by fusing the global feature and the local feature, and the similarity parameter is used to characterize the degree of similarity between the feature vector and the target feature; determining a maximum preset number of exhibits in the exhibit feature library with similarity parameters to the target feature as candidate exhibits; if the similarity parameter of the candidate exhibit is greater than a preset similarity threshold, determining that the target image and the candidate exhibit are successfully matched; if the similarity parameter of the candidate exhibit is greater than the preset similarity threshold, determining that the target image has failed to match, and returning a first prompt message to the terminal device that uploaded the target image, wherein the first prompt message is used to indicate that exhibit information of the exhibit was not found.

[0073] Specifically, in the embodiment of the present application, when performing a matching search in the exhibit feature library, a voting strategy can be constructed to take the average of the top-k scores (i.e., similarity parameters) in each category of exhibits for voting, and select the exhibit category with the highest average score as the top 1 matching category (if the score is greater than a threshold), such as Figure 6 As shown, after calculating the similarity parameter (similarity score) between the feature vector of each exhibit in the exhibit feature library and the target feature corresponding to the target image, the top-3 candidates for each category of exhibits can be taken to calculate the weighted score of the category cls_score = mean (cls_top3_scores). cls_top3_scores is the similarity score of the three most similar stored image vectors of the same category of cultural relics retrieved from the vector library, and cls_score is the weighted score of the category. If the maximum score (Score) of all categories is greater than the preset matching threshold (similarity threshold), the voting decision is made to obtain the category and the corresponding score. Otherwise, "unknown" is returned, that is, the first prompt information mentioned above is returned.

[0074] Through the voting decision-making strategy, the recognition accuracy can be improved in scenes with densely displayed exhibits (such as bronze group exhibitions); and it supports robust matching of multi-view exhibits, which can significantly improve the recognition stability of exhibits that do not exceed the maximum tilt image in the sample library.

[0075] In order to further improve the recognition accuracy and avoid the problem of misidentification between exhibits that have not been stored and other exhibits that are relatively similar to the exhibit library, the embodiment of the present application can also, based on the user misidentification feedback mechanism, perform incremental updates to the blacklist library (historical misidentification list) according to the feedback exhibit photos provided after the target image and the candidate exhibit are successfully matched during the preliminary matching, and dynamically constrain the complex open domain retrieval to an approximate closed domain recognition problem, as follows.

[0076] In some embodiments of the present application, after determining that the target image and the candidate exhibit are successfully matched, the method further includes the following steps: determining a similarity parameter between a target feature of the target image and a target feature of each historical misjudgment image in a historical misjudgment list; if there is a historical misjudgment image in the historical misjudgment list whose second similarity parameter is greater than the first similarity parameter, returning a first prompt message to the terminal device that uploaded the target image, wherein the second similarity parameter is a similarity parameter between the target feature of the target image and the target feature of the historical misjudgment image, and the first similarity parameter is a similarity parameter between the target feature of the target image and a feature vector of the candidate exhibit; if there is no historical misjudgment image in the historical misjudgment list whose second similarity parameter is greater than the first similarity parameter, obtaining exhibit information of the candidate exhibit and sending the exhibit information to the terminal device, wherein the exhibit information includes: an explanation word corresponding to the candidate exhibit.

[0077] In some embodiments of the present application, after sending the exhibit information to the terminal device, the method further includes the following steps: upon receiving a first feedback message sent by the terminal device, determining the target image as a historical misjudgment image, and storing the historical misjudgment image and the target features of the historical misjudgment image in a historical misjudgment list, wherein the first feedback message is used to characterize that the exhibit information does not match the exhibit contained in the target image.

[0078] Specifically, if Figure 7 As shown, after the target image is determined to be successfully matched with the candidate exhibit during the preliminary matching, the target image will be further matched with the historical misjudgment images stored in the historical misjudgment list (blacklist) to determine whether the exhibits in the target image are all on the historical misjudgment list (blacklist). If the exhibit obtained by voting (i.e., the exhibit determined to be successfully matched with the target image and the candidate exhibit) is on the historical misjudgment list (blacklist), the query is intercepted and marked as "unknown exhibit", and the first prompt information is returned; if it is not on the historical misjudgment list (blacklist), the candidate exhibit is considered to be the exhibit in the target image, and the exhibit information of the candidate exhibit, such as the explanation words, is returned to the user terminal.

[0079] In addition, the embodiment of the present application can also construct a hierarchical exhibit classification system and establish standardized guidelines for data collection of exhibits at each level, as follows.

[0080] In some embodiments of the present application, the method also includes: determining the exhibit level corresponding to the exhibit in the target image, wherein the exhibit level includes: museum level, exhibition level, explanation point level, and cultural relic level; obtaining exhibit data corresponding to the exhibit level in the exhibit feature library; matching the exhibit data corresponding to the exhibit level based on global features and local features to obtain exhibit information corresponding to the target image.

[0081] Specifically, if Figure 8 As shown in the figure, a four-level exhibit relationship of "museum-exhibition-explanation point-cultural relics" can be constructed, and standardized guidelines for data collection of exhibits at each level can be established. Compared with single-level exhibits, the recognition accuracy of tourists when photographing exhibits of different granularities (such as single exhibits and display cabinets composed of multiple exhibits) can be significantly improved.

[0082] As an optional implementation, the exhibit feature library further includes: the exhibition location and exhibition time of the exhibit; the method further includes the following steps: determining the shooting location and shooting time of the terminal device that uploaded the target image when shooting the target image; obtaining candidate exhibit data in the exhibit feature library whose exhibition location is within a target area and whose exhibition time is within a target time range, wherein the target area is an area with a preset length as the radius centered on the shooting location, and the target time range is a time range within a preset time length before and after the shooting time; matching is performed in the candidate exhibit data based on global features and local features to obtain exhibit information corresponding to the target image.

[0083] The intelligence and practicality of the recognition system are further enhanced by utilizing spatiotemporal positioning information. By obtaining the GPS coordinates and shooting time of the terminal device, the system is able to narrow the search scope and only consider exhibits on display at a specific time and place, which greatly improves recognition efficiency and reduces unnecessary calculations. For example, if a user takes a photo of an exhibit at two o'clock in the afternoon, the system will prioritize looking for matches in the list of exhibitions for that afternoon, rather than blindly searching the entire database. This technical solution solves the problem of how to quickly locate targets in large-scale databases. By combining spatial and temporal information, the system can quickly lock onto targets in massive amounts of data, reducing the consumption of computing resources and improving recognition response speed, while ensuring the relevance and timeliness of information, providing users with an efficient and accurate recognition experience.

[0084] The embodiments of the present application can more accurately identify exhibits by combining global features and local features for matching, especially under complex backgrounds or low-contrast conditions. The contrast is enhanced by histogram equalization processing, thereby improving the accuracy of recognition. At the same time, the introduction of exhibit hierarchy and spatiotemporal positioning information further improves the accuracy and efficiency of recognition and reduces the possibility of misidentification. In addition, the establishment of a feedback mechanism enables the system to self-correct based on user feedback, continuously optimize the recognition model, and enhance the user experience. This method is not only suitable for museums, but can also be extended to various exhibitions, artwork recognition and other scenarios, and has broad application prospects and practical value.

[0085] This application scheme has a wide range of adaptability to exhibits, and solves the contradiction that related technologies are difficult to simultaneously perform high-precision identification of "layout exhibits" and "independent cultural relics"; it improves the instability of related technologies in identifying cultural relics from different perspectives, which is mainly reflected in the easy occurrence of misidentification at different perspectives; it improves the difficulty of traditional methods in balancing the recognition accuracy of tourists when photographing exhibits of different levels (such as single exhibits and display cabinets composed of multiple exhibits); it solves the problem that related technologies, due to matching in an open domain, make unstocked exhibits easily misidentified as similar stocked exhibits; and it improves the versatility of the system and reduces the maintenance cost of the photo recognition system.

[0086] According to an embodiment of the present application, an embodiment of a museum exhibit identification device is also provided. Figure 9 FIG. 1 is a schematic diagram of a museum exhibit identification device according to an embodiment of the present application. Figure 9 As shown, the device includes:

[0087] An image acquisition module 90 is used to acquire a target image, wherein the target image includes an exhibit to be identified;

[0088] The subject detection module 92 is used to perform single subject detection on the target image to obtain a subject area image in the target image, wherein the subject area image is the portion of the target image containing the exhibit to be identified;

[0089] A feature extraction module 94 is used to extract features from the target image to obtain global features, and to extract features from the subject area image to obtain local features;

[0090] The exhibit matching module 96 is used to perform matching searches in the exhibit feature library based on global features and local features to obtain exhibit information corresponding to the target image. The exhibit feature library contains feature vectors of multiple exhibits, and the feature vectors are determined based on the global features and local features of the exhibits.

[0091] Optionally, performing single-subject detection on the target image to obtain a subject area image in the target image includes: using a single-subject detection model to analyze the target image to obtain a detection frame and a confidence level corresponding to the detection frame, wherein the detection frame is used to select the exhibit as the subject in the target image; cropping the target image according to the coordinate information of the detection frame with the highest confidence level to obtain a subject area image; and when the contrast of the subject area image is lower than a preset contrast threshold, performing histogram equalization processing on the subject area image to enhance the contrast of the subject area image, wherein the contrast of the enhanced subject area image is not lower than the preset contrast threshold.

[0092] Optionally, matching in the exhibit feature library based on global features and local features includes: determining a similarity parameter between a feature vector of each exhibit in the exhibit feature library and a target feature corresponding to the target image, wherein the target feature includes: a global feature and a local feature, or a fusion feature obtained by fusing the global feature and the local feature, and the similarity parameter is used to characterize the degree of similarity between the feature vector and the target feature; determining a maximum preset number of exhibits in the exhibit feature library with similarity parameters to the target feature as candidate exhibits; if the similarity parameter of the candidate exhibit is greater than a preset similarity threshold, determining that the target image and the candidate exhibit are successfully matched; if the similarity parameter of the candidate exhibit is greater than the preset similarity threshold, determining that the target image has failed to match, and returning a first prompt message to the terminal device that uploaded the target image, wherein the first prompt message is used to indicate that exhibit information of the exhibit was not found.

[0093] Optionally, after determining that the target image and the candidate exhibit are successfully matched, the method further includes: determining a similarity parameter between a target feature of the target image and a target feature of each historical misjudgment image in a historical misjudgment list; if there is a historical misjudgment image in the historical misjudgment list whose second similarity parameter is greater than the first similarity parameter, returning a first prompt message to the terminal device that uploaded the target image, wherein the second similarity parameter is a similarity parameter between the target feature of the target image and the target feature of the historical misjudgment image, and the first similarity parameter is a similarity parameter between the target feature of the target image and a feature vector of the candidate exhibit; if there is no historical misjudgment image in the historical misjudgment list whose second similarity parameter is greater than the first similarity parameter, obtaining exhibit information of the candidate exhibit and sending the exhibit information to the terminal device, wherein the exhibit information includes: an explanation word corresponding to the candidate exhibit.

[0094] Optionally, after sending the exhibit information to the terminal device, it also includes: when receiving a first feedback message sent by the terminal device, determining the target image as a historical misjudgment image, and storing the historical misjudgment image and the target features of the historical misjudgment image in a historical misjudgment list, wherein the first feedback message is used to characterize that the exhibit information does not match the exhibit contained in the target image.

[0095] Optionally, the museum exhibit recognition device is also used to: determine the exhibit level corresponding to the exhibit in the target image, wherein the exhibit level includes: museum level, exhibition level, explanation point level, and cultural relic level; obtain exhibit data corresponding to the exhibit level in the exhibit feature library; and match the exhibit data corresponding to the exhibit level based on global features and local features to obtain exhibit information corresponding to the target image.

[0096] Optionally, the exhibit feature library further includes: the exhibition location and exhibition time of the exhibit; the museum exhibit identification device is further used to: determine the shooting location and shooting time of the terminal device that uploaded the target image when shooting the target image; obtain candidate exhibit data in the exhibit feature library whose exhibition location is within the target area range and whose exhibition time is within the target time range, wherein the target area range is an area range with a preset length as the radius centered on the shooting location, and the target time range is a time range within a preset time length before and after the shooting time; based on global features and local features, matching is performed in the candidate exhibit data to obtain exhibit information corresponding to the target image.

[0097] It should be noted that the various modules in the above-mentioned museum exhibit identification device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0098] It should be noted that the museum exhibit identification device provided in this embodiment can be used to perform Figure 2 The museum exhibit identification method shown in the figure, therefore, the relevant explanations and descriptions of the above-mentioned museum exhibit identification method are also applicable to the embodiments of the present application and will not be repeated here.

[0099] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following museum exhibit identification method by running the computer program: obtaining a target image, wherein the target image contains the exhibit to be identified; performing single-subject detection on the target image to obtain a subject area image in the target image, wherein the subject area image is the portion of the target image containing the exhibit to be identified; performing feature extraction on the target image to obtain global features, and performing feature extraction on the subject area image to obtain local features; performing matching retrieval in an exhibit feature library based on the global features and the local features to obtain exhibit information corresponding to the target image, wherein the exhibit feature library contains feature vectors of multiple exhibits, and the feature vectors are determined based on the global features and the local features of the exhibits.

[0100] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the museum exhibit identification method described in each embodiment of the present application: obtaining a target image, wherein the target image contains the exhibit to be identified; performing single-subject detection on the target image to obtain a subject area image in the target image, wherein the subject area image is the portion of the target image containing the exhibit to be identified; performing feature extraction on the target image to obtain global features, and performing feature extraction on the subject area image to obtain local features; performing matching retrieval in an exhibit feature library based on the global features and the local features to obtain exhibit information corresponding to the target image, wherein the exhibit feature library contains feature vectors of multiple exhibits, and the feature vectors are determined based on the global features and the local features of the exhibits.

[0101] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0102] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0104] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0105] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0107] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for identifying museum exhibits, characterized in that: include: Acquiring a target image, wherein the target image includes an exhibit to be identified; Performing single subject detection on the target image to obtain a subject area image in the target image, wherein the subject area image is a portion of the target image containing the exhibit to be identified; Performing feature extraction on the target image to obtain global features, and performing feature extraction on the subject area image to obtain local features; Based on the global features and the local features, a matching search is performed in an exhibit feature library to obtain exhibit information corresponding to the target image, wherein the exhibit feature library contains feature vectors of multiple exhibits, and the feature vectors are determined based on the global features and local features of the exhibits.

2. The museum exhibit identification method according to claim 1, characterized in that: Performing single subject detection on the target image to obtain a subject area image in the target image includes: Using a single-subject detection model, the target image is analyzed to obtain a detection frame and a confidence score corresponding to the detection frame, wherein the detection frame is used to frame the exhibit as the subject in the target image; Cropping the target image according to the coordinate information of the detection frame with the highest confidence to obtain the subject area image; When the contrast of the main body area image is lower than a preset contrast threshold, histogram equalization processing is performed on the main body area image to enhance the contrast of the main body area image, wherein the contrast of the enhanced main body area image is not lower than the preset contrast threshold.

3. The museum exhibit identification method according to claim 1, characterized in that: Matching in the exhibit feature library based on the global features and the local features includes: Determining a similarity parameter between the feature vector of each exhibit in the exhibit feature library and a target feature corresponding to the target image, wherein the target feature includes: the global feature and the local feature, or a fusion feature obtained by fusing the global feature and the local feature, and the similarity parameter is used to represent the degree of similarity between the feature vector and the target feature; Determine the exhibits with the largest preset number of similarity parameters between the exhibit feature database and the target feature as candidate exhibits; When the similarity parameter of the candidate exhibit is greater than a preset similarity threshold, determining that the target image is successfully matched with the candidate exhibit; When the similarity parameter of the candidate exhibit is greater than a preset similarity threshold, it is determined that the target image matching fails, and a first prompt message is returned to the terminal device that uploaded the target image, wherein the first prompt message is used to indicate that the exhibit information of the exhibit was not found.

4. The museum exhibit identification method according to claim 3, characterized in that: After determining that the target image successfully matches the candidate exhibit, the method further includes: Determining a similarity parameter between a target feature of the target image and target features of each historical misjudgment image in a historical misjudgment list; If there is a historical misjudgment image in the historical misjudgment list whose second similarity parameter is greater than the first similarity parameter, returning the first prompt information to the terminal device that uploaded the target image, wherein the second similarity parameter is a similarity parameter between a target feature of the target image and the target feature of the historical misjudgment image, and the first similarity parameter is a similarity parameter between the target feature of the target image and a feature vector of the candidate exhibit; When there is no historical misjudgment image whose second similarity parameter is greater than the first similarity parameter in the historical misjudgment list, the exhibit information of the candidate exhibit is obtained and sent to the terminal device, wherein the exhibit information includes: an explanation word corresponding to the candidate exhibit.

5. The museum exhibit identification method according to claim 4, characterized in that: After sending the exhibit information to the terminal device, the method further includes: Upon receiving the first feedback message sent by the terminal device, the target image is determined to be the historical misjudgment image, and the historical misjudgment image and the target features of the historical misjudgment image are stored in the historical misjudgment list, wherein the first feedback message is used to characterize that the exhibit information does not match the exhibit contained in the target image.

6. The museum exhibit identification method according to claim 1, characterized in that: The method further comprises: Determining an exhibit level corresponding to the exhibit in the target image, wherein the exhibit level includes: museum level, exhibition level, explanation point level, and cultural relic level; Acquire exhibit data corresponding to the exhibit level in the exhibit feature library; According to the global features and the local features, matching is performed in the exhibit data corresponding to the exhibit level to obtain exhibit information corresponding to the target image.

7. The museum exhibit identification method according to claim 1, characterized in that: The exhibit feature database further includes: the exhibit location and exhibition time of the exhibit; the method further includes: Determining a shooting position and a shooting time of a terminal device that uploaded the target image when shooting the target image; Acquire candidate exhibit data from the exhibit feature database, wherein the exhibit location is within a target area and the exhibit time is within a target time range, wherein the target area is an area with a preset radius centered at the shooting location, and the target time range is a time range within a preset time period before and after the shooting time; Matching is performed in the candidate exhibit data based on the global features and the local features to obtain exhibit information corresponding to the target image.

8. A museum exhibit identification device, characterized in that: include: An image acquisition module, configured to acquire a target image, wherein the target image includes an exhibit to be identified; a subject detection module, configured to perform single subject detection on the target image to obtain a subject area image in the target image, wherein the subject area image is a portion of the target image containing the exhibit to be identified; A feature extraction module is used to extract features from the target image to obtain global features, and to extract features from the subject area image to obtain local features; An exhibit matching module is used to perform matching retrieval in an exhibit feature library based on the global features and the local features to obtain exhibit information corresponding to the target image, wherein the exhibit feature library contains feature vectors of multiple exhibits, and the feature vectors are determined based on the global features and local features of the exhibits.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is configured to run a program stored in the memory, wherein the program, when running, executes the museum exhibit identification method according to any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the museum exhibit identification method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the museum exhibit identification method according to any one of claims 1 to 7 are implemented.

Images