A layered adaptive lost and found intelligent matching method and system

Abstract

The present application relates to the technical field of multi-modal matching, and particularly relates to a layered self-adaptive lost and found intelligent matching method and system.A layered self-adaptive lost and found intelligent matching method is provided, which adopts a multi-modal layered matching strategy, uses a feature with small calculation overhead but high discrimination to quickly and roughly screen, combines time and location constraints to eliminate unreasonable candidates, and finally performs fine fusion of images and texts on a smaller candidate set.Through layered screening, space-time constraints and adaptive weight fusion, the present application can quickly and accurately match lost and picked-up items in a large-scale candidate library, reduce calculation amount and improve matching accuracy and interpretability.

Description

Technical Field

[0001] This invention relates to the field of multimodal matching technology, and in particular to a hierarchical adaptive intelligent matching method and system for lost and found. Background Technology

[0002] As lost and found services on campus gradually move online, students and faculty members typically upload photos of items, brief descriptions, and information about the time and location when the items were lost or found, hoping that the system can automatically match them and provide candidate results.

[0003] For example, the lost and found method and system (CN104796861A) uses a location-based node allocation method, which determines the target lost and found node by using the information of the location where the lost item was picked up. Users can query lost item information and retrieve the item according to the code. This solution relies only on text description and geographical location, without utilizing image information; it lacks an intelligent matching mechanism, requiring users to manually filter; and it cannot handle vague descriptions and differences in image quality.

[0004] The intelligent public transport lost and found system (CN119848457A) uses AI algorithms to generate a candidate set of lost item descriptions and performs matching based on description similarity and trajectory consistency analysis. However, this solution relies heavily on textual semantics and makes insufficient use of image information; it lacks a hierarchical filtering mechanism, resulting in a large computational load for full-database matching; and its fixed-weight fusion method cannot adapt to the differences in characteristics among different categories of items.

[0005] The lost and found platform based on image matching (“One Item for One Item”) (Wei Yuwei, Yuan Wenli, Han Jiawei, et al. “One Item for One Item” - Implementation of Image Matching Function for Lost and Found [J]. Electronic Production, 2021(12):40-41,86.DOI:10.3969 / j.issn.1006-5059.2021.12.016.) adopts simple image similarity comparison. Such schemes only consider single image features (such as color histograms), have poor robustness; do not combine time and space constraints, resulting in a high mismatch rate; lack multimodal fusion, and cannot handle the situation of missing information.

[0006] However, in real-world scenarios, data sources are often inconsistent: some records only contain images without text descriptions; others only contain text without clear images; the same item exhibits significant differences under different devices and lighting conditions, and shooting angles, occlusions, and reflections can also lead to instability in color, shape, and texture features. Matching methods relying solely on a single modality or feature are prone to failure under these influences. Text-related information also exhibits instability. On the one hand, users have inconsistent descriptive habits, and the same item may have multiple names, with descriptions potentially vague or containing spelling errors; on the other hand, OCR has significant recognition errors on low-resolution, curved surfaces, or reflective materials, making text similarity difficult to use directly as a reliable basis. This makes it difficult for solutions relying solely on textual clues to achieve stable results under the generally low image quality and descriptive level of campus environments.

[0007] Furthermore, while time and location information is frequently recorded, it is often merely displayed in many systems and not incorporated into the matching process in a computationally efficient manner. In reality, most lost and found incidents on campus occur within a specific time window and a limited spatial area. Ignoring the constraints of time difference and geographical distance can easily allow clearly unreasonable candidates to be included in subsequent comparisons, increasing false matches, significantly raising computational costs, and slowing down response times, especially during peak periods such as exam weeks and the start of the school year.

[0008] Different categories of items also have different dependencies on features. Books are generally more sensitive to text and pattern structure, wallets rely more on color and material texture, keys are mainly based on shape and metal texture, and water cups may have both shape and a small amount of text. If fixed feature weights are used, it is difficult to maintain overall stability in campus data with mixed categories. Summary of the Invention

[0009] To overcome the problems of existing lost and found systems, such as reliance on a single modality, low computational efficiency, poor adaptability, poor interpretability, and insufficient utilization of spatiotemporal information, this invention proposes a hierarchical adaptive intelligent matching method for lost and found. This method employs a multimodal, multi-level adaptive matching strategy to achieve fast and accurate matching of lost items with found items in a large-scale candidate database.

[0010] This invention proposes a hierarchical adaptive intelligent matching method for lost and found, comprising the following steps: S1. Collect lost and found information and lost and found notice information. After standardizing the image, text, time and location information, store them in the lost and found collection and the lost and found notice collection respectively. Among them, the location information is unified into calculable location coordinates, and the location confidence level L is recorded for each piece of information. When the location information is missing or is a vague location description, the location confidence level L is set to a smaller value or a preset default value. S2. Based on the coarse screening feature vectors of each lost and found notice in the lost and found notice set, establish an approximate nearest neighbor ANN index, and use the coarse screening feature vector of each lost and found notice as a query vector to retrieve the ANN index, and return the Top N candidate lost and found notices to form a coarse screening set D1 corresponding to the lost and found notice. S3. Calculate the spatiotemporal correlation between the candidate lost and found notices and their corresponding lost and found information in the coarse screening set D1, and filter the coarse screening set D1 based on the spatiotemporal correlation to obtain the spatiotemporal set D2; wherein, the spatiotemporal correlation is obtained by weighted fusion of temporal correlation and spatial correlation, and the spatial correlation is weighted or attenuated based on the distance between the coordinates of the two parties and combined with the position credibility L of the two parties, so that the contribution of spatial correlation decreases when the position credibility decreases; S4. Traverse the spatiotemporal set D2, calculate the modal similarity between lost and found information and candidate lost and found notices in multiple specified modalities, and fuse the modal similarity according to the modal weights to obtain a fusion score; S5. Sort the candidate lost and found notices in the spatiotemporal set D2 based on the fusion score, and select the top K candidate lost and found notices with the highest scores as the matching object set. S6. Send a matching notification to the publisher based on the set of matching objects and collect feedback, or trigger a supplementary information request and / or manual review when the maximum fusion score is lower than the matching threshold. S7. When the update triggering condition is met, the matching threshold and / or the modal weight are adaptively updated based on the feedback; wherein, the update triggering condition includes: the cumulative number of lost and found items processed reaches the first threshold N1, or the cumulative number of correct matching feedback obtained reaches the second threshold N2.

[0011] Preferably, the ANN index is a combination index of inverted file index and product quantization (IVF+PQ), or an HNSW index, and the retrieval distance metric is L2 distance and / or cosine distance.

[0012] Preferably, the coarse screening feature vector includes at least two of the following: HSV color histogram features, the percentage of pixels at the edge of the target object, the image aspect ratio, and color moment features.

[0013] Preferably, the spatiotemporal correlation is obtained by weighted fusion of temporal correlation Rt and spatial correlation Rs, satisfying: R = Rt + Rs, where + =1; When lost and found information or candidate lost and found notices lack time information, let Rt take the default value; When lost and found information or candidate lost and found notices lack location information or cannot be parsed into location coordinates, let Rs take the default value.

[0014] Preferably, the time correlation Rt is a function that decreases monotonically with time difference Δt; The time correlation Rt satisfies one of the following: ; or ;

[0015] in, The difference in recording time between lost and found notices and lost item announcements. This is a time-scale parameter.

[0016] Preferably, the spatial correlation Rs is a function that decreases monotonically with spatial distance d, and the spatial correlation Rs is weighted or decreased based on the location confidence L. The spatial correlation Rs satisfies one of the following: ; or ; Where Lq represents the location reliability of the lost and found information, and Ld represents the location reliability of the candidate lost and found notices. Let g(Lq,Ld) be a spatial scale parameter, a non-negative function of Lq and Ld, and g(Lq,Ld) does not increase when Lq or Ld decreases, and d is the distance between the location recorded in the lost and found notice and the location recorded in the lost and found notice.

[0017] Preferably, in step S3, the coarse set D1 is filtered based on a spatiotemporal threshold, and candidate missing item notices with a spatiotemporal correlation not less than the spatiotemporal threshold are retained to form a spatiotemporal set D2.

[0018] Preferably, the fusion score is a weighted fusion result of the similarity of each modality, and the weight of each modality is non-negative and satisfies the normalization constraint; and the modality similarity includes at least one of the following: color similarity, local feature matching similarity, texture similarity, and text semantic similarity.

[0019] Preferably, the adaptive update of the matching threshold is based on the deviation between the mismatch rate of the most recent M matches and the target mismatch rate; the adaptive update of the modality weights is based on the contribution ratio of each modality to the fusion score in the most recent R correct match feedbacks and is normalized.

[0020] The present invention proposes a hierarchical adaptive intelligent matching system for lost and found, comprising: a memory, a processor, and a computer program stored in the memory and executable by the processor; when the processor executes the computer program, it implements the hierarchical adaptive intelligent matching method for lost and found.

[0021] The advantages of this invention are: (1) The present invention proposes a hierarchical adaptive intelligent matching method for lost and found items. It adopts a multimodal hierarchical matching strategy, firstly using features with low computational cost but high discriminative power for rapid coarse screening, then combining time and location constraints to eliminate unreasonable candidates, and finally performing fine fusion of images and text on a small candidate set. The present invention achieves fast and accurate matching of lost items and found items in a large-scale candidate library through hierarchical screening, spatiotemporal constraints and adaptive weight fusion, reducing the amount of computation and improving the matching accuracy and interpretability.

[0022] (2) A hierarchical strategy of “color coarse screening → spatiotemporal constraints → multimodal fine matching” is adopted, and the approximate nearest neighbor retrieval index and Top-N candidate control are used in the coarse screening stage to significantly reduce the subsequent high-cost calculations. By synthesizing spatiotemporal correlation through time windows and spatial thresholds, “soft filtering” is performed on the coarse screening results, which can reduce noisy candidates without sacrificing recall; when information is missing, it can be downweighted to maintain the stable operation of the system.

[0023] (3) This invention supports multiple fusion strategies such as weighted summation / maximum value / average, and can combine category knowledge to set prior weights, which are then further refined by dynamic weights; it is more adaptable and transferable than baseline methods with a single fusion strategy. On different item categories (such as books, notebooks, wallets, cups, etc.), this invention can highlight dominant features such as color or texture according to category, reducing cross-class bias.

[0024] (4) The coarse screening stage of this invention uses a standardized near nearest neighbor retrieval index, which is beneficial for horizontal expansion and cross-device deployment; the number of candidates can be flexibly configured according to resource and latency targets.

[0025] (5) The present invention can also automatically adjust the weights and thresholds of different features according to the matching feedback results. The dynamic weights and thresholds can be updated in small steps through user feedback, which has online adaptive capability and can continuously improve the hit rate in real use of the system, thereby significantly improving the matching speed and stability while ensuring accuracy. Attached Figure Description

[0026] Figure 1 This is a flowchart of a hierarchical adaptive intelligent matching method for lost and found proposed in this invention. Figure 2 A schematic diagram for constructing the coarse sieve set D1; Figure 3 This is a schematic diagram of spatiotemporal constraints and filtering. Figure 4 This is a schematic diagram of multimodal fusion screening; Figure 5 This example compares the method of the present invention with comparison method 3 at different update coefficients. Figure 6The performance improvement of the method of the present invention compared to the comparative method 3 is shown for different update coefficients; Figure 7 A comparison of inference time for different methods. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0028] like Figure 1 As shown, the present invention proposes a hierarchical adaptive intelligent matching method for lost and found, which includes the following steps S1-S7.

[0029] S1. Collect lost and found notices and lost item notices, and after standardizing the text, store them in the lost and found collection and the lost item notice collection respectively.

[0030] The standardized processing method for the text is as follows: adjust the image size and angle, and extract the HSV histogram; combine the text description and OCR results to perform text parsing and extract the text content T. std Unify time and space formats, and record location reliability. The HSV histogram is... It can be directly reused in subsequent similarity calculation steps.

[0031] Specifically, standardized copywriting processes include: Image normalization processing: By parsing the EXIF ​​(Exchangeable image file format) orientation information in the image file, the rotation angle of the image is determined, and the image data is rotated to ensure it is displayed upright. The image is scaled so that the shorter side is scaled to a set pixel threshold while maintaining the aspect ratio. The image is then converted to the HSV color space, and the histograms of the H, S, and V channels are extracted and normalized separately so that the sum of the histograms of each channel is 1. Text standardization processing: The user description and OCR (Optical Character Recognition) results are merged to form the original text. The original text is then segmented, corrected, replaced with synonyms, and filtered for stop words to obtain the text content. ; Time and location standardization: Time is standardized to UTC timestamps. Location using latitude and longitude (WGS-84), and record the location reliability. When the published information is missing a location or the location is vaguely described, the credibility of the location is adjusted. Take the smaller value; when the position has explicit coordinates or can be resolved into coordinates, let Take the larger value, which will be used for weighted attenuation in subsequent spatial correlation calculations.

[0032] Traverse the lost and found collection and find matching lost and found notices for each item based on the following steps S2-S5.

[0033] S2. Based on visual feature retrieval, find lost and found notices that match with lost and found notices to form a coarse set D1.

[0034] Specifically, refer to Figure 2 Coarse feature vectors were extracted from lost and found notices and missing item notices respectively. and the collection of lost and found notices Establish an approximate nearest neighbor index based on IVF+PQ, referred to as the lost item index; use the coarse screening feature vector of lost and found items. The top-N lost and found notices are retrieved from the lost and found index as query vectors and used as candidate lost and found notices, forming a coarse screening set D1.

[0035] in: ; This is the normalized 32-dimensional (bin) hue histogram, which is the result of normalizing the 32-dimensional (bin) hue histogram to a sum of 1. This is the normalized 32-dimensional (bin) saturation histogram, which is the result of normalizing the 32-dimensional (bin) saturation histogram to a sum of 1. The proportion of Canny edge pixels to the total number of pixels in the image; The aspect ratio of the image is specifically the ratio of the number of pixels in the length direction to the number of pixels in the width direction. and The features are assembled in a preset order to form a coarse-screened feature vector.

[0036] Corresponding to Lost and Found When using the query vector to retrieve the index for lost and found items, the top-N closest lost and found notices can be retrieved based on L2 distance (Euclidean distance) as candidates, forming a coarse-screen set. N can specifically take the value 200.

[0037] Specifically, after collecting lost and found notices and standardizing the text, a coarse feature vector for each lost and found notice can be calculated. They are then associated and stored in the Lost and Found collection and the Found Notice collection.

[0038] S3. Traverse the coarsely sieved set D1, calculate the lost and found notice Q and the candidate lost and found notices. Time correlation Spatial correlation The spatiotemporal correlation was obtained by weighted fusion. The spatiotemporal relevance of the screening and lost and found services to the lost and found services is greater than or equal to the set spatiotemporal threshold. The candidate lost and found notices constitute a spatiotemporal set D2. In specific implementation, it can be set... .

[0039] Specifically, refer to Figure 3 The formula for calculating spatiotemporal correlation is: ; ;or, ; ;or, ; in, This indicates the lost and found notice and candidate lost and found notices. (The i-th lost and found notice in the coarse set D1) Spatiotemporal correlation, and These represent the lost and found notice (Q) and the candidate lost and found notice, respectively. Temporal and spatial correlations and These are the time and space weights, respectively, and their sum is 1. Specifically, they can be set to... , ; The time recorded for the lost and found Q With candidate lost and found notices Recording time The absolute value of the difference For time-scale hyperparameters; in specific implementation, it can be set as follows: In hours Set to 24; d represents the location recorded in the lost and found notice and the candidate lost and found notices. The distance between recorded locations can be calculated using the Haversine formula. This is a spatial scale hyperparameter; in practice, the unit of d can be set to meters. Set to 500; when lost and found Or candidate lost and found notice When time information is missing, let When lost and found Or candidate lost and found notice When location information is missing or cannot be resolved to latitude and longitude coordinates, let . Let Lq be the spatial scale parameter, Ld be the location reliability of the lost and found information, and Ld be the location reliability of the candidate lost and found notices. g(Lq,Ld) is a non-negative function of Lq and Ld, and g(Lq,Ld) does not increase when Lq or Ld decreases. Specifically, the following two settings can be used: (1) Product type: ; (2) Shortest plank type: min ;

[0040] Location confidence can be calculated using existing confidence calculation methods based on multimodal information.

[0041] S4. Traverse the spacetime set D2, comparing the lost and found notice Q with candidate lost and found notices. Modal similarity is calculated across five modalities: color histogram, local features, texture, text semantics, and spatiotemporal correlation. The modal similarities are then weighted and fused to obtain a fusion score. ,like Figure 4 As shown.

[0042] (1) Color histogram similarity The image is parsed using HSV, and the 32-dimensional histograms of the H and S channels are extracted and concatenated to obtain a 64-dimensional histogram vector; Lost and Found Q and candidate lost and found notices. The 64-dimensional histogram vectors are denoted as follows: The 64-dimensional histogram is normalized to satisfy: ; Color histogram similarity is defined as histogram intersection similarity: ; in, and The color histogram is non-negative and L1 normalized, such that the sum of all bins (dimensions) is 1. A larger value indicates that the colors are more similar.

[0043] (2) Local feature similarity (SIFT + RANSAC) right and Extract SIFT keypoint sets from the images respectively. , And the corresponding set of descriptors. Descriptor matching yields a set of matching pairs. (The number of matching pairs is denoted as) ), and use RANSAC to perform geometric consistency verification to obtain the set of interior points. (The number of interior points is denoted as) ).

[0044] 1) Matching percentage (normalized to) ): ; in, express and The matching percentage, and They are respectively and The number of points.

[0045] 2) Matching distance normalization term: For all inlier matching pairs Take its SIFT descriptor L2 distance Calculate the average descriptor distance of interior points. : ; And in the same search (Lost and Found) candidate set Normalize internally (to avoid differences) (Inconsistent scale) ; in, for The normalized value; To prevent extremely small positive numbers with a denominator of 0.

[0046] 3) Local feature similarity is defined as: ; in, For Q and Local feature similarity, A larger value indicates that the local structures are more similar; For Q and The matching percentage. If Then let .

[0047] (3) Texture similarity (LBP + chi-square distance) For Q and The LBP texture histogram is extracted from the image and normalized to obtain... , Both satisfy the condition that their histogram sum is 1. The chi-square distance is defined as: ; And in the candidate set Normalization is performed internally: ; For Q and Chi-square distance, for The normalized value.

[0048] Texture similarity is defined as: ; in, .

[0049] (4) Text semantic similarity

[0050] Lost and Found Text and Lost and Found Notice The texts are input into a pre-trained Chinese sentence vector encoding model to obtain sentence vectors. , Text similarity is defined as: ; in For cosine similarity, .

[0051] Spatiotemporal correlation similarity directly uses spatiotemporal correlation. And define spatiotemporal modal similarity as: ; The fusion score is obtained by weighting and fusing the similarities of each modality: ; in, ; and In practice, the corresponding weights can be set as follows:

[0052] S5. For each candidate lost item notice in the spatiotemporal set D2 Fusion score Sort in descending order and select the top scorers. A lost and found notice as a lost and found item notice The set of matching objects .

[0053] S6. The set of matching objects obtained in step S5 As a lost and found The matching list, denoted as The Used for subsequent notification sending and feedback records.

[0054] Get the set of matching objects Maximum fusion score: ; judge Is it not less than the matching threshold? : like If the error occurs, a "Requires additional information" message will be output, and a request for additional information will be sent to the publishers in the matching list List(Q); and / or List(Q) will be transferred to manual review. like If a match is found, a matching notification is sent to the publishers in List(Q), and the publisher's feedback information is recorded.

[0055] Specifically, the feedback information from the publisher may include: confirming / denying a match, providing supplementary images, supplementary text descriptions, and supplementary time and location of loss / recovery, etc.; the feedback information is written into List(Q) for subsequent statistics and parameter updates.

[0056] S7. Feedback-based adaptive update, the update object includes the matching threshold. With weight .

[0057] When the set triggering conditions are met, based on the feedback information collected in step S6, the matching threshold is adjusted. and / or modal weights Update.

[0058] (1) Matching threshold Update Statistics recently The mismatch rate of each lost and found item is calculated, and the matching threshold is updated based on the mismatch rate.

[0059] Specifically, a "mismatch" is defined as: for a certain lost and found item... If all matching objects in its matching list List(Q) return an error, then the lost item will be returned to the claiming authorities. Counted as 1 mismatch; most recent The number of mismatches in the lost and found items is: The false match rate is: ; Let the target mismatch rate be... Then the fine-tuning coefficient is: ; The formula for updating the matching threshold is: ; in, The updated matching threshold, The existing matching threshold; For fine-tuning, the specific value can be the difference between the mismatch rate of the most recent M lost and found items and the target mismatch rate. When all matching objects of a mistakenly found item are reported as incorrect, the lost and found item is recorded as a mismatch item, and the mismatch rate is the proportion of mismatch items among the M lost and found items. M and the target false rejection rate are set values, specifically, M=50 and the target mismatch rate is 10%.

[0060] (2) Modal weights Feedback Update When the triggering condition is met, based on the most recent The correct feedback updates the modal weights. The trigger condition can be any of the following: Processed since the last weight update A lost and found item; or, the cumulative points gained since the last weight update. This is the second correct match feedback. N1 and N2 are both set values.

[0061] "Correct match feedback" means that at least one matching object in the matching list of a lost and found item is reported as correct.

[0062] Weight updates include the following sub-steps: St1, Sample Construction: From the most recent In each correct match feedback, the "matching object with the correct feedback and the highest fusion score" is selected to form a paired sample. , 1≤r≤R. and These are the lost and found notices and the missing item notices for the r-th correctly matched object, respectively.

[0063] St2, Calculate the modal contribution value: due to the modal scores For paired samples Calculate modes Contribution percentage: ; in, For paired samples The weights corresponding to mode j during the calculation process, For paired samples Similarity on modality j; For paired samples The weights corresponding to mode u during the calculation process For paired samples Similarity on mode u.

[0064] In the above formula, For the first The fusion score of each paired sample under the weights before the update (including...) (used to avoid a denominator of 0), thus... Approximately 1, representing the percentage contribution of each modality to the fusion score of that sample. For the most recent... A number of paired samples { The average value of the feedback contribution is obtained by calculating the values ​​of 1 ≤ r ≤ R. ; in, For paired samples Computational mode The percentage of contribution, For modality The feedback contribution value.

[0065] Step 3: Update weights and normalize: That is, in this step, the weights are first calculated. Update value Then to Normalization yields the updated weights , making The formula is expressed as follows: ; ; in, To update the coefficients, for example, 0.02.

[0066] The following describes the hierarchical adaptive intelligent matching method for lost and found (hereinafter referred to as the method of this invention) proposed in this invention, with reference to specific embodiments.

[0067] This embodiment also constructs the following three comparison methods: Comparison Method 1: Match lost and found items with lost and found notices by analyzing image HSV histograms, item outlines, and colors; Comparison Method 2: First, the lost and found notices are matched with the image HSV histogram, item outline and color to obtain candidate lost and found notices for physical items; then, the spatiotemporal correlation between physical items and candidate lost and found notices is calculated, and the final matching object (i.e. lost and found notice) is selected based on the spatiotemporal correlation from high to low. Comparison Method 3: Compared with the method of the present invention, the matching threshold of Comparison Method 3 is... and modal weights All values ​​are fixed.

[0068] In this embodiment, 100 lost and found notices were matched against each other in the same database, and the experimental results were statistically analyzed. The experimental results of each comparison method and the method of the present invention are shown in Table 1 below.

[0069] In this embodiment, the item categories include books, notebooks, wallets, cups, etc.

[0070] Table 1: Statistical Analysis of Experimental Results

[0071] In Table 1: Precision@1 means that only 1 lost and found notice is selected as the final matching target for the precision analysis; Precision@5 means that 5 lost and found notices are selected as the final matching target for the precision analysis; and Precision@10 means that 10 lost and found notices are selected as the final matching target for the precision analysis.

[0072] In this implementation, the accuracy rate is the percentage of exact matches. An exact match means that at least one of the matched items found is accurate.

[0073] As shown in Table 1, in the Precision@5 experiment, the accuracy of the method of the present invention was improved from 0.4000 to 0.7800 compared with the comparative method 1 (which is also the most commonly used prior art), an improvement of about 95%; and the method of the present invention is superior to the comparative method 2.

[0074] In the Precision@10 experiment, the accuracy of the method of the present invention was improved from 0.2000 to 0.3900 compared with the comparative method 1, an improvement of about 95%; and the method of the present invention is superior to the comparative method 2.

[0075] Therefore, in real-world scenarios where multiple candidates (with a large Top-N) need to be returned, the method of this invention can maintain higher accuracy more stably, reduce the proportion of irrelevant results, and improve user filtering efficiency.

[0076] In this embodiment, the coefficients are also being updated. Compare the method of the present invention and the comparative method 3 on different values. When the weight ratio is 0, the method of this invention is equivalent to comparison method 3, i.e., fixed weight. In this case, the accuracy of the two methods overlaps in each experiment. With... As the system gains more trust in dynamic weights, the advantages of the method in this invention become increasingly apparent. When taking intermediate values, the transition is smooth, avoiding instability caused by drastic weight changes.

[0077] from Figure 5 , Figure 6It can be seen that in the Precision@5 experiment, when At approximately 0.32, a significant improvement of over 2% has been achieved; the maximum improvement is +15.28%; in the Precision@10 experiment, when At approximately 0.21, a significant improvement of over 2% was achieved; the maximum improvement was +38.20%. In these two experiments, When the value is approximately 0.5, the accuracy of the method of the present invention shows an accelerated increase.

[0078] It is evident that dynamic weights can adaptively allocate the importance of color, texture, local features, and text according to category and sample characteristics, solving the problem of fixed weights "favoring one aspect at the expense of another" in different categories and scenarios, and possessing room for sustainable optimization.

[0079] Figure 7 Further comparison of the inference time of each method shows that the query time of the method of the present invention is only 0.0178 ms, which is about 2.3% longer than that of the comparison method 1 (0.0174 ms), but achieves an improvement of about 95% in Precision@5; at the same time, it is lower than that of the comparison method 2 (0.0192 ms) and the comparison method 3 (0.0190 ms).

[0080] As can be seen, this invention employs a hierarchical strategy of "color coarse screening → spatiotemporal constraints → multimodal fine matching," and uses an approximate nearest neighbor retrieval index and Top-N candidate control in the coarse screening stage, significantly reducing subsequent high-cost computations. While ensuring improved accuracy, this invention maintains almost no impact on end-to-end latency, making it more suitable for online retrieval and large-scale deployment.

[0081] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0082] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0083] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A hierarchical adaptive intelligent matching method for lost and found, characterized in that, Includes the following steps: S1. Collect lost and found information and lost and found notice information. After standardizing the image, text, time and location information, store them in the lost and found collection and the lost and found notice collection respectively. Among them, the location information is unified into calculable location coordinates, and the location confidence level L is recorded for each piece of information. When the location information is missing or is a vague location description, the location confidence level L is set to a smaller value or a preset default value. S2. Based on the coarse screening feature vectors of each lost and found notice in the lost and found notice set, establish an approximate nearest neighbor ANN index, and use the coarse screening feature vector of each lost and found notice as a query vector to retrieve the ANN index, and return the Top N candidate lost and found notices to form a coarse screening set D1 corresponding to the lost and found notice. S3. Calculate the spatiotemporal correlation between the candidate lost and found notices and their corresponding lost and found information in the coarse screening set D1, and filter the coarse screening set D1 based on the spatiotemporal correlation to obtain the spatiotemporal set D2; wherein, the spatiotemporal correlation is obtained by weighted fusion of temporal correlation and spatial correlation, and the spatial correlation is weighted or attenuated based on the distance between the coordinates of the two parties and combined with the position credibility L of the two parties, so that the contribution of spatial correlation decreases when the position credibility decreases; S4. Traverse the spatiotemporal set D2, calculate the modal similarity between lost and found information and candidate lost and found notices in multiple specified modalities, and fuse the modal similarity according to the modal weights to obtain a fusion score; S5. Sort the candidate lost and found notices in the spatiotemporal set D2 based on the fusion score, and select the top K candidate lost and found notices with the highest scores as the matching object set. S6. Send a matching notification to the publisher based on the set of matching objects and collect feedback, or trigger a supplementary information request and / or manual review when the maximum fusion score is lower than the matching threshold. S7. When the update triggering condition is met, the matching threshold and / or the modal weight are adaptively updated based on the feedback; wherein, the update triggering condition includes: the cumulative number of lost and found items processed reaches the first threshold N1, or the cumulative number of correct matching feedback obtained reaches the second threshold N2.

2. The hierarchical adaptive intelligent matching method for lost and found as described in claim 1, characterized in that, The ANN index is a combination of inverted file index and product quantization index IVF+PQ, or an HNSW index, and the retrieval distance metric is L2 distance and / or cosine distance.

3. The hierarchical adaptive intelligent matching method for lost and found as described in claim 1, characterized in that, The coarse screening feature vector includes at least two of the following: HSV color histogram features, the percentage of pixels at the edge of the target object, the image aspect ratio, and color moment features.

4. The hierarchical adaptive intelligent matching method for lost and found as described in claim 1, characterized in that, The spatiotemporal correlation is obtained by weighted fusion of temporal correlation Rt and spatial correlation Rs, satisfying: R = Rt + Rs, where + =1; When lost and found information or candidate lost and found notices lack time information, let Rt take the default value; When lost and found information or candidate lost and found notices lack location information or cannot be parsed into location coordinates, let Rs take the default value.

5. The hierarchical adaptive intelligent matching method for lost and found as described in claim 1 or 4, characterized in that, The time correlation Rt is a function that decreases monotonically with time difference Δt; The time correlation Rt satisfies one of the following: ； or ; in, The difference in recording time between lost and found notices and lost item announcements. This is a time-scale parameter.

6. The hierarchical adaptive intelligent matching method for lost and found as described in claim 1, characterized in that, The spatial correlation Rs is a function that decreases monotonically with spatial distance d, and the spatial correlation Rs is weighted or decreased based on the location confidence L. The spatial correlation Rs satisfies one of the following: ； or ; Where Lq represents the location reliability of the lost and found information, and Ld represents the location reliability of the candidate lost and found notices. Let g(Lq,Ld) be a spatial scale parameter, a non-negative function of Lq and Ld, and g(Lq,Ld) does not increase when Lq or Ld decreases, and d is the distance between the location recorded in the lost and found notice and the location recorded in the lost and found notice.

7. The hierarchical adaptive intelligent matching method for lost and found as described in claim 1, characterized in that, In step S3, the coarse set D1 is filtered based on the spatiotemporal threshold, and candidate missing item notices with spatiotemporal correlation not less than the spatiotemporal threshold are retained to form the spatiotemporal set D2.

8. The hierarchical adaptive intelligent matching method for lost and found as described in claim 1, characterized in that, The fusion score is a weighted fusion result of the similarity of each modality, and the weight of each modality is non-negative and satisfies the normalization constraint; and the modality similarity includes at least one of the following: color similarity, local feature matching similarity, texture similarity, and text semantic similarity.

9. The hierarchical adaptive intelligent matching method for lost and found as described in claim 1, characterized in that, The adaptive update of the matching threshold is based on the deviation between the mismatch rate of the most recent M matches and the target mismatch rate; the adaptive update of the modality weight is based on the contribution ratio of each modality to the fusion score in the most recent R correct match feedbacks and is normalized.

10. A hierarchical adaptive intelligent matching system for lost and found, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable by the processor; When the processor executes the computer program, it implements the hierarchical adaptive intelligent matching method for lost and found as described in any one of claims 1-9.

Images