Method and system for ordering reference images

Abstract

The present invention relates to a method (P) for ordering images (Iref) of a reference base (Bref) on the basis of criteria comprising an identification datum, the method (P) filtering the images (Irec) of a search base (Brec) according to the identification datum and comparing the filtered images (Irec) with the images (Iref) of the reference base (Bref) so as to generate, for each reference image (Iref) – search image (Irec) pair, a similarity score, the generated similarity score being modified according to the confidence index of the common class of the identification datum and the metadatum for characterising the image (Irec) to be searched.

Description

[0001] Title of the invention: METHOD AND SYSTEM FOR ORDERING REFERENCE IMAGES

[0002] The present invention relates to the field of ordering images in order to be able to present them in order of relevance. More particularly, the invention concerns the ordering of images whose subject is a physical element in the context of a search for a given physical element.

[0003] In a first example, it may involve finding an object among lost objects and for this purpose ordering images of lost objects. In a second example, it may also involve searching for a person among referenced individuals and for this purpose ordering images of the referenced individuals. Other examples of technical contexts may benefit from the invention.

[0004] Traditionally, in decision-making or decision-support systems, aimed at identifying objects or people and based on comparison algorithms providing scores at the end of these comparisons, the decision to validate the comparisons is taken during a final examination step (called adjudication) on the basis of a list of candidates provided by the algorithm. Similarity scores resulting from the comparisons made by the system are calculated by the comparison algorithm, between the search data and the reference data. A threshold on the similarity score is generally established in order to filter sufficiently relevant records and the system will return to the operator of this decision-making system a list of records from the reference database whose comparison scores of the reference data with the search data are higher than the threshold.The operator must then make a decision based on the list received.

[0005] The efficiency and productivity of the decision system therefore depends on the speed with which the operator finds and confirms valid matches between records in the reference database and records in the research database with which the research data have been submitted for comparison.

[0006] To this end, the order of presentation of the results is in descending order of score and the operator will confirm or deny the matches by following this order, hence the importance of the order of presentation and of not excluding by overly restrictive filtering in the reference base of "true positive" candidates for the productivity and efficiency of the decision system. Thus, the processes filtering images on the basis of search criteria then comparing the filtered images to a reference image, by means of an analysis by neural networks in particular, in order to determine a score and order the images according to their score, are not always satisfactory because the filtering sometimes excludes the image that should have been kept.

[0007] It is also possible to avoid this pitfall by not filtering and analyzing the entire database directly, but productivity is then lower because the processing time is longer and the probability is greater of having a larger number of candidates to order, which reduces the probability that the ordering will bring out the truly relevant candidate(s) in the best ranks.

[0008] It should be noted that in the context of lost baggage searches, carried out by airlines or airports, investigation time and performance are essential and are now critical, generating significant costs (search and reimbursement fees) for lesser results.

[0009] The invention aims to solve all or part of these problems by proposing a method for ordering reference images which quickly and reliably produces a list of the most relevant images with respect to the formulated search.

[0010] According to one aspect of the invention, there is provided a method for ordering reference images recorded in a reference base, said method being implemented by computer and comprising the following steps:

[0011] - association with each search image of at least one characterization metadata belonging to a class, said search images and characterization metadata forming a search base,

[0012] - receiving a search request for a physical element sought including as search criterion an identification data item for said physical element sought belonging to the same class as the at least one characterization metadata item, said class of the identification data item being associated with a confidence index characterizing the link between said physical element sought and said identification data item;

[0013] - filtering the search base according to the identification data of said physical element sought by comparison with the characterization metadata of the same class so as to determine images to be searched for associated with said query; if the filtering of the search base determines at least two images to be searched for associated with said query, said images to be searched not all corresponding to the same physical element: - comparing each reference image of all or part of the reference base with said at least two images to be searched for by means of an image comparison algorithm so as to generate for said reference image a similarity score between said reference image and each of said at least two images to be searched for;

[0014] - modify the similarity score generated based on the confidence index of the said same class, common to the identification data and the characterization metadata specific to the image to be searched, in particular by weighting;

[0015] - order all or part of said reference images by decreasing modified similarity score.

[0016] This method, through the development and consideration of a confidence index of the identification data resulting from the query, makes it possible to take into consideration, through this notion of link between the physical element sought and the identification data, the value of the search criterion, i.e. its relevance, its importance. The modification of the similarity score a posteriori, according to this confidence index, makes it possible in particular not to exclude images on the basis of a criterion of lesser reliability. It therefore makes it possible to efficiently and quickly order, in order of relevance, all or part of the reference images, said reference images representing a physical element, on the basis of a selection of search images, said search images from the search base representing a physical element, according to search criteria.Said ordered reference images can then be presented to a user, such as an auction operator, as a decision-making aid, or for example be used to automate the selection, in particular if the modified similarity score of the first ordered reference image is significantly different from the modified similarity score of the second ordered reference image, the method then being able to continue, including without human intervention, in one embodiment, with the sending, to the applicant for example, of the physical element sought now found, corresponding to that imaged in the first ordered reference image, that is to say the one which was selected.

[0017] Advantageously, the determination of said confidence index of a class is prior to the reception of the request, which allows rapid processing, the confidence indices by class being for example recorded in a table, and established by adjustments during a pilot phase.

[0018] Advantageously, the confidence index of a class depends on said class and on a relative quantity of physical elements that said class covers, relative to another class, which makes it possible to compare the classes with each other according to the strength of the link (relationship) which unites their members to the underlying search criterion and according to their discriminating power. Thus, for example, the determination of the confidence index of a class makes it possible to take into consideration, in a combined manner:

[0019] • The fact that the identification data considered points to a greater or lesser number of physical elements.

[0020] • The greater or lesser probability that, among these physical elements to which the identification data points, the physical element sought is actually present.

[0021] In fact, the fewer the number of physical elements to which the identification data points and the greater the probability that the physical element sought is part of these physical elements, the greater the confidence index characterizing the link between the identification data and the physical element sought and therefore the greater the value of the search criterion.

[0022] The relative quantity can be statistically assessed during a pilot calibration phase.

[0023] Advantageously, the method includes a step of normalizing the similarity scores, in particular before using the confidence indices to modify the similarity scores, which in particular makes it possible to threshold and compare the scores with each other.

[0024] Advantageously, the confidence index of the identification data class is a function of the search query, which makes it possible, for example, to favor the family name class when the name is rare compared to a very common name, and in this case if the confidence index had been predetermined prior to the query (table) it is then modified.

[0025] Advantageously, the filtering of the search base is carried out by Boolean filtering of the characterization metadata, which allows a rapid search on a search base prepared for this purpose by the characterization metadata associated with each image. In addition, filtering from new metadata by computer analysis of the image, using a neural network, can be carried out so as to extract information not present in the characterization metadata, which allows, if this analytical filtering is carried out upstream on the search base and provides quality information, to add the new metadata to the characterization metadata.

[0026] Advantageously, the image comparison algorithm uses a neural network. Advantageously, the modification of the similarity score is carried out by weighting or by increment / decrement, which allows easy implementation, the value of the weighting factor or of the increment or of the decrement being a function of the confidence index of the common class of the identification data and of the characterization metadata specific to the image to be searched, which allows a highlighting of the reference images Iref for which the relationship with the search criteria is the strongest (highest confidence index) and, in a simplified embodiment, the weighting weight can be equal to the confidence index.

[0027] In one embodiment, the physical element is an object or a person.

[0028] Advantageously, several characterization metadata belonging to distinct classes are associated with all or part of the search images, which makes it possible to filter the search base on the basis of several search criteria, as long as the identification data and the characterization metadata of the base share the same classes respectively.

[0029] Advantageously, the at least one characterization metadata is specific to a context of acquisition of said research image, which allows at least one of the characterization metadata, if there are several, to be in particular non-visual.

[0030] Advantageously, at least one reference metadata is associated with each recorded reference image, including in particular a reference metadata of the same class as the identification data of said physical element sought, which makes it possible to pre-filter the reference base, in particular on the basis of the search criterion defined by the identification data of the same class.

[0031] In one embodiment, the method according to the invention comprises a step of pre-filtering the reference base by comparing the identification data of said physical element sought with the referencing metadata of the same class so as to select a part of the reference base forming a list of candidate reference images for comparison with said at least two images to be searched, which makes it possible to carry out the comparison step only on a reduced list of reference images of the reference base.Advantageously, the identification data is a search start date for said physical element sought and the at least one reference metadata of the reference image is an acquisition date of the reference image, said reference metadata being specific to a context of acquisition of said reference image, the pre-filtering of the reference base being a function of said search start date, which makes it possible to filter the reference base according to, for example, the date of loss of the baggage so as to exclude from the list of candidate reference images the images of baggage photographed at the lost baggage storage facility before the search start date, the latter corresponding in this case to the date of loss of the baggage.

[0032] Advantageously, the similarity score, specific to the reference image and image to be searched pair, is modified according to a difference between the values ​​of the referencing metadata of the reference image and the characterization metadata of the image to be searched of the same class, which makes it possible to take into account metadata characterizing the similarity between the physical elements which are not taken into consideration in the comparison algorithm, either intrinsically because they are not of a visual nature, or because the conditions of their illustration in the images are not guaranteed if they are visual (for example if they designate a visual detail too small to be taken into account reliably by the comparison algorithm, or too dependent on the shooting conditions).

[0033] In one embodiment, the at least one metadata of the image is included in the image or is recorded separately in the database which contains it, which allows easy and rapid filtering, the recording in the image, that is to say in particular in the image file, being already used for example for the acquisition dates.

[0034] Advantageously, the at least one characterization or referencing metadata of the search or reference image, respectively, is extracted by computer analysis of said image, in particular by means of neural networks, which makes it possible, whether for the referencing metadata of the images in the reference database, or for the characterization metadata of the images in the search database, to create classes a posteriori, in particular on the fly, which makes it easy to enrich the search criteria, even if these criteria had not, for example, been initially planned, in particular as long as these criteria are visual. In such a case, it is particularly interesting to modify accordingly the weighting or the increment / decrement used when modifying the similarity score so as to take into consideration in their values ​​a score representative of the quality of the extraction of the metadata by computer analysis.

[0035] In one embodiment, the physical element is a person, the reference images being images of referenced individuals and the search images being images of unidentified individuals acquired by video surveillance in different places, the common class of identification data and characterization metadata is for example a date, a place or a size, which allows for an identification data such as a place of interest at a given time, a size of the person sought, and respectively the at least one characterization metadata such as a date of acquisition of the search image having as subject the individual subject of the search image, a size of the individual subject of the search image, a place of acquisition of the search image,to be able to compare at least one characterization metadata with at least one search criterion and in particular for metadata of the location of interest type at a given time to calculate a distance between the location of acquisition of the search image and the location of interest, or even the difference between said given time and the date of acquisition of the search image having as its subject the individual subject of the search image, knowing that the location of interest is for example the location of a crime or a location indicated in the context of a testimony. The locations and dates are in particular metadata (characterization or respectively referencing) specific to the context of acquisition, that is to say capture, of the image with which the metadata is associated.,

[0036] Likewise, advantageously, the at least one reference metadata is a size of the individual subject of the reference image, which makes it possible to pre-filter and, above all, to evaluate for each reference image the differences in quantitative data between these reference metadata and the identification data of the same class, or the characterization metadata of the image to be searched for of the same class, and in particular to serve for the modification of the similarity scores.

[0037] In one embodiment, the physical element is an object, the object being a piece of luggage, the reference images being images of stored luggage and the search images being images of enrolled luggage acquired during their enrollment, which makes it possible to use the images acquired during the enrollment of the luggage, i.e. the photographs captured jointly with the registration of the luggage, in particular by means of baggage drop-off kiosks or image acquisition arches arranged downstream of the registration in the baggage sorting system, to, following a request for loss of luggage, identify on the basis of search criteria the enrollment images to be searched in the reference database,so as to then carry out the comparison between the images to be searched selected by filtering on the basis of the search criteria and to order the images among the reference base: that is to say among the photographs of the non-attributable baggage (that is to say in particular without label) stored in the warehouses containing this non-attributable lost baggage, in order of relevance so as to transmit the results of the ordering to an adjudication agent in charge of monitoring customer requests. This application to baggage allows in particular a central analysis of the data without necessarily having to conduct manual investigations warehouse by warehouse.,

[0038] Advantageously, the search start date is that of a last registration of said baggage sought or of a last flight made with said baggage, which makes it possible to pre-filter the reference base.

[0039] Advantageously, the common class of identification data and characterization metadata is a size of the baggage, a weight of the baggage, a color of the baggage, a flexible or rigid nature of the baggage or a data of enrollment of the baggage, which allows for the former to use non-contextual metadata, easily extracted by image analysis, and for the latter, metadata specific to the context, to provide information, almost exclusively non-visual, much more limiting, because specific to the baggage or to the person who enrolled it, this class of identification data allowing effective filtering of the search base containing this metadata.

[0040] Advantageously, the given class characterization metadata for enrolling the baggage is a receipt number, a depositor identifier associated with the baggage, a flight reservation number, a flight number or an enrollment date (corresponding to the date of acquisition of the image of the baggage in the search base), which makes it possible in particular to filter the images in the search base according to this information, that is to say according to, for example, the surname of the depositor of the baggage or his frequent traveler number, and to formulate one or more queries to the bases.

[0041] In one embodiment, several images of the reference base correspond to different views of the same baggage or several images of the search base correspond to different views of the same baggage, which makes it possible to increase the chances of having pairs of images from directly comparable views for the same baggage present in the two bases (and therefore to have better similarity scores), then to present to the operator this baggage as a candidate for the search result with a rank calculated according to the best available score corresponding to the pair of images having carried it out, to present to him both this pair of images but also the other images of the two bases corresponding to this candidate baggage in order to assist his adjudication task thanks to additional images.

[0042] According to another aspect of the invention, there is provided a system for ordering reference images recorded in a reference base comprising:

[0043] - an association module which associates with each search image of a search base at least one characterization metadata belonging to a class;

[0044] - a network communication device;

[0045] - a processing device that executes computer-readable instructions that, when executed by the processing device, cause the processing device to:

[0046] - receiving, by means of the network communication device, a search request for a physical element sought comprising as search criterion an identification data item for said physical element sought belonging to the same class as the at least one characterization metadata item, said class of the identification data item being associated with a confidence index characterizing the link between said physical element sought and said identification data item;

[0047] - filtering the search base according to the identification data of said physical element sought by comparison with the characterization metadata of the same class so as to determine images to be searched for associated with said query; if the filtering of the search base determines at least two images to be searched for associated with said query, said images to be searched not all corresponding to the same physical element:

[0048] - comparing each reference image of all or part of the reference base to said at least two images to be searched by means of an image comparison algorithm so as to generate a similarity score between said reference image and each of said at least two images to be searched;

[0049] - modify the similarity score generated based on the confidence index of the said same class, common to the identification data and the characterization metadata specific to the image to be searched;

[0050] - ordering all or part of said reference images by decreasing modified similarity score. This system has the same advantages as the ordering method according to the invention, the ordering system according to the invention being in particular capable of implementing the ordering method according to the invention in its various embodiments.

[0051] According to another aspect of the invention, a computer program is provided comprising instructions adapted to the implementation of each of the steps of the scheduling method according to the invention when said program is executed on a computer.

[0052] According to another aspect of the invention, there is provided a means of storing information, removable or not, partially or totally readable by a computer or a microprocessor comprising code instructions of a computer program for the execution of each of the steps of the method according to the invention.

[0053] In the attached drawings, given as non-limiting examples: Figure 1 represents a flowchart of the scheduling method according to one embodiment of the invention; Figure 2 represents a flowchart of the scheduling method adapted to the case of searching for luggage according to one exemplary embodiment of the invention; Figure 3 represents the architecture of the reference image scheduling system according to one embodiment of the invention; and Figure 4 is a schematic block diagram of a scheduling system comprising an information processing device for implementing one or more embodiments of the invention.

[0054] The invention can be applied in different contexts, Figure 1 is intended to illustrate the scheduling method P according to the invention in a global and synthetic manner. The method P schedules, all or part, of the reference Iref images recorded in a reference Bref base, the method P comprises the following steps - association E1 with each search image Irec of a search Brec base of at least one characterization metadata belonging to a class, - receiving E2 a search request REQ for a physical element sought comprising as search criterion an identification data item of said physical element sought belonging to the same class as the at least one characterization metadata item, said class of the identification data item being associated with a confidence index characterizing the link between said physical element sought and said identification data item;- filter E3 the Brec search base according to the identification data of said physical element sought by comparison with the characterization metadata of the same class so as to determine images to be searched for associated with said query; if COND the filtering of the search base determines at least two images to be searched for associated with said query, said images to be searched not all corresponding to the same physical element;

[0055] - comparing E4 each reference image of all or part of the reference base to said at least two images to be searched by means of an image comparison algorithm so as to generate for said reference image a similarity score between said reference image and each of said at least two images to be searched;

[0056] - modify E5 the similarity score generated based on the confidence index of the common class of the identification data and the characterization metadata specific to the image to be searched, in particular by weighting or by increment / decrement;

[0057] - order E6 all or part of said reference images by decreasing modified similarity score.

[0058] The step of determining a confidence index for the class of the identification data is not illustrated here because this step is preferably carried out upstream. The confidence indices of a given class then have a pre-established fixed value (for example following a pilot phase) and their values ​​are independent of the query. However, this step can also be carried out in two stages, so the confidence index specific to the given class is initialized with the pre-established value in a first stage, then in a second stage, between step E2 of receiving the query and step E3 of filtering the search base (or at any time between step E2 of receiving and step E5 of modifying the scores) the value of said confidence index is updated, which presents in particular an advantage when the confidence index specific to the class of the identification data depends on the search query,that is to say that for the same class of identification data the value of the confidence index specific to this identification data will not be the same depending for example on the “value” of the identification data itself in the query (for example a common surname versus a rare surname), or on one of its attributes such as the number of identification data of the same class in the query (for example the number of baggage receipts submitted). The determination step can also be carried out at once between step E2 of receiving the query and step E3 of filtering the search base (or at any time between step E2 of receiving and step E5 of modifying the scores),which allows in a single step a refined consideration of the characteristics specific to each physical element to be searched for. The method of carrying out the determination step is advantageously a combination of these three alternatives chosen according to the class of the data.,

[0059] The Brec research database therefore contains records (or records in English), here we are talking about recordings in the form of images knowing that these images can also come from video streams. These Irec records from the Brec research database are associated ab initio (in particular for metadata specific to the acquisition context, for example the acquisition date, etc.) or after an image analysis (for example a size of the physical element) with characterization metadata, this characterization metadata is used during the E3 filtering of the research database, filtering, which is preferably Boolean but can also be accompanied by an image analysis by neural network, in particular if certain metadata were not yet associated with all the Irec research images (such as a new visual characteristic that would have been mentioned as a search criterion).If said image analysis of the database makes it possible to determine and associate the new metadata with all the Irec images of the search database before filtering, then a new class of characterization metadata is created and used by the Boolean filtering. The characterization metadata can be associated with each image in that they are directly included in the image file or they can be recorded separately in the Brec search database, in particular centralized, for example in the form of a table, a combination of the two modes also allowing secure redundancy of the information. It should also be noted that the nature of the storage can also depend on the metadata itself, some of which may be sensitive, especially among the metadata specific to the acquisition context and require special conditions, some of this data can advantageously be encrypted.The characterization metadata used during the E3 filtering of the search database are chosen based on the identification data that constitute the search criteria imposed by the query and for each search criterion a confidence index is determined and associated with the class of the identification data. This confidence index illustrates the value of the search criterion, by characterizing the strength of the link between said physical element sought and said identification data. This confidence index will thus be representative of the selective value of said search criterion in particular so as to represent the level of certainty that the physical element selected on the basis of this search criterion corresponds to the physical element sought.In fact, a low confidence index implies a low probability that the physical element selected by filtering in the search base corresponds to the one sought, while a high confidence index implies a high probability that the physical element selected by filtering in the search base corresponds to the one sought. This confidence index thus makes it possible to extend the search in a controlled manner because this confidence index will be taken into consideration in the step of a posteriori modification of the similarity scores. A table including the confidence index specific to each class of metadata can in particular be stored, their calibration values ​​resulting for example from a pilot optimization phase.

[0060] The records selected after E3 filtering of the Brec research database: i.e. the images to be searched, are then used to be compared E4 with the records (here also in the form of images) of the Bref reference database using an image comparison algorithm (also called a matching algorithm in English). The objective of this comparison is to be able to determine whether records in the research database correspond to records in the reference database. During the E4 comparison, the metadata (whether characterization or possibly reference), as such, are not used because the comparison algorithm is essentially based on the analysis of the image pixels or on the use of trained neural networks for image comparison.At the output of this comparison step E4, a similarity score is generated for each reference image and image pair to be searched.

[0061] It is during step E5 that this similarity score is modified according to the confidence index specific to the image to be searched Irec, this confidence index being that of the class common to the identification data and to the characterization metadata since it is on the basis of the identification data in question that the image to be searched in question was selected. This modification E5 of the similarity score is preferably carried out by weighting, which makes it possible, by its commutative nature, to be insensitive to the order of multiple modification steps, since in the case where the search image was selected on the basis of multiple identification data, that is to say that it fulfilled the criterion in question each time, then several modifications are carried out according to the confidence index specific to said identification data, not all the search images having necessarily fulfilled all the criteria.The weighting factors (or increment / decrement), specific to each class and linked to the confidence indices, are preferably stored in a table, or can be calculated according to a formula of which one of the parameters is the confidence index.As the reference images Iref are compared to several images to be searched Irec (for example several shots, or several times the same search image compared to different search criteria titles), at least two similarity scores are calculated for each reference image since a calculation of this score is carried out for each reference image - image to be searched pair, it is therefore advantageous to include, during the ordering step E6, for said reference image Iref a selection of the maximum score among the scores of said reference image Iref - image to be searched Irec pairs, so as to be able to order in an optimized manner and so that each reference image Iref is presented only once (a single rank) to the adjudication agent, in the order given by the ordering step E6.The same logic for selecting the maximum score by groups of images (multi-views) Iref of the same physical element applies so as, for example, not to show the different views of the same physical element at different ranks in the ordered list delivered to the adjudication agent. Alternatively, several requests can be expressed, each request comprising only one identification data item, the consolidation by selection of the maximum score of the modified similarity scores of each reference image being carried out in the ordering step E6.It is therefore understood that the modification E5 of the scores according to the confidence index therefore changes the order of presentation of the candidates (by decreasing scores) between an ordering at the end of the comparison step E4 and the ordering E6, which allows the adjudication agent to process the correspondences returned in decreasing order of scores by seeing in priority possible correspondences having higher probabilities of being true correspondences ('true positives' in the language of decision systems) and which consequently allows to improve the productivity and the efficiency of the decision process as a whole.

[0062] In the remainder of the description, the ordering method is described in two context examples, one in which the physical element to be searched for is an object, in particular luggage, and another in which the physical element to be searched for is a person.

[0063] Thus, the exemplary embodiment shown in Figure 2 is placed in the context of the loss of baggage by a traveler. The ordering method therefore aims to help an adjudication agent identify the baggage declared lost by a traveler among unattributable baggage (without a baggage registration label, the label attached to the baggage when it was registered having been lost or poorly attached, or not attached for example) stored in one or more warehouses. Photographs (for example on a scale so as to have the weight) of this unattributable baggage are taken and constitute the reference base while the photographs taken during the registration of the baggage before travel constitute the search base. It will be understood that this embodiment finds an interesting application for air transport.

[0064] During check-in, baggage enrollment systems, as described in patent application FR1902969, capture Irec images of each baggage item as well as other information associated with said baggage (for example, the identity of the baggage dropper, his frequent flyer number, the reservation number, the number of the flight taken, the weight of the baggage, etc.) constituting in particular characterization metadata specific to the acquisition context recorded in the search database, the set of Irec images and metadata constituting the Brec search database, each Irec image being associated with at least one characterization metadata, and in particular with at least one characterization metadata specific to the acquisition context.

[0065] Furthermore, the airline or airport is organized to collect all baggage that has lost its identification tag and is therefore unattributable, cannot be transported to its intended destination, and is therefore lost. This lost baggage converges on one or more locations where it will be stored and photographed; these are the reference Iref images that constitute the reference Bref database. It is therefore understood that this reference Bref database is dynamic here, depending on the arrival of lost baggage and the collection of baggage whose owner has been found.

[0066] When a traveler presents himself to make a claim by reporting that he has lost baggage, a list of identification data as search criteria is then established. A search query for the sought (lost) baggage including as search criterion the identification data in question is then sent to the scheduling system. In the case where certain information communicated by the traveler is not available (for example the date of purchase of the baggage), that is to say not extractable from the search base, and therefore not corresponding to a characterization metadata class, this information is preferably not part of the query (which does not mean that the information will not be used but not within the framework of the method described here) or is excluded from the query, then reformulated, during the reception step E2.The search query REQ for a physical element sought includes as search criterion one or more identification data of said physical element sought belonging to classes, for example, the traveler who declares his baggage lost indicates that the lost baggage is red, rigid and that the context of the loss of the baggage is such a flight on such a date, associated with his name. The descriptive characteristics provided by the traveler can be positive or negative. Advantageously, if the traveler submits during his declaration a photograph of the baggage that he declares lost, this photograph can be used in step E3 of filtering the search base so as not to use only the photograph taken by the traveler, which could be of lower quality, during the comparison step but also the search images Irec (as well as the multiple other views of the baggage of said image Irec) which would have had the best scores in particular in step E3 of filtering.The Brec search base is filtered in step E3 according to the identification data of the baggage sought by comparison with the characterization metadata of the same class in order to determine images to be searched for associated with said request.

[0067] Possible classes include:

[0068] - the traveler's first and last name, linked to the traveler and their namesakes and therefore covering all of their luggage known to the system during the data retention period,

[0069] - the reservation number (which can be linked to several people such as a family or a group and therefore to all their luggage, including the lost luggage),

[0070] - the flight number (linked to all passengers on the flight and all their baggage),

[0071] - the frequent flyer number,

[0072] - the baggage receipt number...

[0073] This generally results in several images to be searched that do not correspond to the same baggage being selected unless, for example, the traveler has attached their baggage receipt. Indeed, identification data such as the name implies the selection of baggage of a namesake, the frequent flyer number includes all the baggage that they may have checked in on their previous flights and sometimes several baggage receipts are attached because the traveler does not know which one corresponds to the lost baggage, and the other information provided does not allow a single one to be distinguished. The condition COND according to which at least two images to be searched for Irec have been selected, and therefore associated with the said physical element sought, is therefore very generally met even in the case of thresholding.And when this condition is not fulfilled (not shown in the figures for reasons of clarity) this means that only one Irec image from the Brec search base has been selected, then all that remains is to make a comparison between each reference image of all or part of the reference base with said selected Irec search image by means of an image comparison algorithm so as to generate for each reference image a similarity score between said reference image and the image to be searched; then to order all or part of said reference images by decreasing modified similarity score.

[0074] The fact that a confidence index is subsequently used makes it possible not to have to exclude search criteria, the filtering will therefore not be over-discriminatory and will not eliminate "true positives", each selected Irec image to be searched for being associated with the metadata justifying its inclusion in this list of images to be searched for.

[0075] The confidence index of the class specific to an identification data item is determined in particular according to the quality of this class to describe the lost baggage in a unique manner and therefore the lower the confidence index the less discriminating it will be, broadening the list of candidates: images to be searched, hence the interest in using them in parallel. Advantageously, the confidence index of the class of the identification data item is a function of the search query, so for example, for a group of receipts corresponding to all the passenger's baggage enrolled (including the lost baggage), the confidence index of the "receipt number" class will be all the lower the number of receipts in the group of receipts.

[0076] The "frequent flyer number" (i.e. the passenger's unique customer number for the airline) covering all checked baggage during trips made by this traveler with the airline during the data retention period, for example 3 months, the confidence index of the "frequent flyer number" class will therefore be, for example, higher than that specific to the surname, because it is more discriminating than the latter.

[0077] The step of determining these confidence indices specific to each class is therefore preferentially dependent on the REQ request, so as to make their value as representative as possible of the link between the baggage sought and said identification data.

[0078] The use of these confidence indices makes it possible to take into consideration the fact that the systems for taking pictures of baggage at registration are not installed in all airports at all check-in baggage registration points since in such a case, for example, the frequent traveler number would be used to obtain Irec search images of different pieces of baggage of the traveler acquired during previous registrations in equipped airports, this baggage being able to be more or less numerous and the related confidence index being lower than that which would have been specific to the receipt of the baggage linked to the flight following which the baggage was lost.

[0079] The use of various identification data makes it possible in particular to be robust to the incomplete coverage of the airport network or to the failure of a network acquisition device or even to the poor quality of certain images in the search base, since it is thus possible to select images of past enrollment, in other airports, not directly linked to the journey during which the baggage was lost.

[0080] The comparison step E4 generates similarity scores for each reference image - image to be searched pair, then in step E5 these similarity scores are modified according to the class specific to the image to be searched (which justified its selection) in question.

[0081] Advantageously, the comparison step E4 is carried out on the basis of a three-dimensional image, knowing that if the search images Irec or reference images Iref are two-dimensional, then a three-dimensional representation is previously modeled to generate said three-dimensional image respectively, for the image to be searched on the basis of the multiple characterization Irec images specific to said enrolled baggage acquired according to different views, and for the reference image on the basis of the reference Iref images of the baggage without a label acquired according to different views at the storage location. This embodiment makes it possible to filter and compare only on the basis of the three-dimensional representations, that is to say by pairs of “objects”.

[0082] The modification E5 is carried out here by weighting, for example the weighting factors are positive and the weighting indices specific to the high confidence indices can therefore be greater than 1 while the weighting factors specific to the low confidence indices can then be less than 1. Alternatively, the weighting factors can be limited between 0 and 1. The weighting factors, for the sake of simplification, can be equal to the confidence indices, their calibration and possible update (determination in two stages) then being defined accordingly. It is easily understood that when an image to be searched has been selected on the basis of several identification data:

[0083] - either their weighting factors can be multiplied (or respectively added in the case of an increment / decrement) between them

[0084] - or, preferably, the weighting factor specific to the highest confidence index will be used so as to select the strongest relationship, i.e. the closest (strength of the link), the most promising (with the strongest discriminating power), so as to only carry out the operational modification calculation once for each similarity score of the image to be searched for (for each pair image to be searched for - reference image).

[0085] Then for each reference image a selection of the maximum modified score among the modified scores (because there are as many scores as there are images to search) is preferentially carried out to robustify the ordering and avoid the appearance of duplicates consisting of the appearance of the same object, but appearing on different reference images, at different ranks.

[0086] The method may also include a thresholding step, for example incorporated into the ordering step E6 so as to order, and therefore present only the reference images whose final modified similarity score would be greater than a predetermined threshold, alternatively the presentation to the adjudication agent may only show a predetermined number of reference images, which amounts to replacing the minimum threshold with a maximum rank.

[0087] It should be mentioned that the reference Bref database can advantageously contain metadata, called referencing metadata, this metadata may have been added to the image file ab initio (for example date of acquisition of the reference image) or for example associated a posteriori after an image analysis (for example color of the luggage).

[0088] If for each Iref image of the reference Bref database a given class referencing metadata is available and this given class is common to that of a characterization metadata, then this referencing metadata can also be used. Advantageously the referencing metadata includes values, for example consider a metadata which would be 'damaged zipper' it could have as value: yes, no and N / A if the information could not be extracted (image of the zipper not usable etc.), in particular by analysis of the Iref reference images using neural networks.

[0089] Optional steps related to SEO metadata are shown in dotted lines in Figure 2.

[0090] Advantageously, referencing metadata is associated at the EA step with the reference images, which then allows, for example if an acquisition date class metadata is available, at an EPF pre-filtering step (by Boolean filtering in particular) to select from the reference images Iref of the Bref reference base only the non-attributable baggage (without label) having an acquisition date later than or equal to the date of the flight that caused the loss of the baggage, also called the search start date. At the end of this EPF pre-filtering step, a restricted list of candidate reference images is therefore generated, and the comparison at step E4 will therefore be carried out on this restricted list only.

[0091] If, in the absence of all the images in the reference database, at least some of the Iref images in the reference Brief database are associated with a given class reference metadata and this given class is common to that of a characterization metadata, then this reference metadata can also be used.

[0092] Thus, in addition or in parallel, in an EA step (presented here following the modification step E5 but which could also take place between the comparison step E4 and the modification step E5 or be part of the score modification step E5) of adjustment of the similarity scores, the similarity scores of each reference image are adjusted according to the identity of their value, their similarity, or the numerical difference of their values ​​in the two Bref, Brec databases. This allows, in particular for visual metadata, and for example not specific to the acquisition context, such as color, not to use them as a filter in the EPF step (to avoid losing true positives) but to use them to raise the similarity score of the luggage sharing the same color.This embodiment is particularly interesting when an identification data of the same class has been provided in the REQ request, and in particular in the case of the “surname” class; in fact when the referencing metadata specific to this “surname” class manages to be associated with the reference Iref image (either manually by an operator when taking the photographs who associates the surname as referencing metadata with the image (because he read it inside the baggage or on an external label or by image analysis using in particular an optical character recognition algorithm, for example applied to an area of ​​the reference image identified as being a cartridge of the baggage) this makes it possible to increase the similarity score as a function of the proximity between the referencing metadata (the “detected” name on the reference Iref image) and the characterization metadata (the “enrolled” name).

[0093] In another variant of application of the method according to the invention, another method of searching for baggage, carried out for example in addition to the operating mode previously described, consists of applying the method by reversing the bases. Thus, the database of non-attributable baggage (without label), constituted in the convergence locations of this baggage, here constitutes the search database (and no longer the reference database). The REQ request remains unchanged except that here the identification data belonging to classes not specific to the acquisition context: such as dimensions of the baggage, color, flexible or rigid nature, etc. take on more meaning, since the characterization metadata specific to the other classes are precisely not provided, the baggage label having been lost, and therefore do not filter anything.After E3 filtering, the images to be searched are submitted for comparison E4 against the reference base, comprising the images of the enrolled baggage (shots acquired at the baggage enrollment points in the airports). The rest of the process remains unchanged.

[0094] In the second example context, the physical element to be searched for is a person, this embodiment is not illustrated in a dedicated manner because it can be read with the support of the figures present.

[0095] In this context of implementation of the method according to the invention, for example in the case of searching for a person who has committed an offense in the context of an investigation, the REQ query(s) are formed on the basis of witness requisitions and the search criteria include identification data such as the location of the offense, the size (known or presumed) of the person sought. For this type of characterization metadata class, the confidence index will be, respectively, for example, decreasing the more the distance between the location of interest and the location of the video surveillance camera used to acquire the recording increases, decreasing the more the difference between the presumed size and that estimated on the search Irec image increases.

[0096] The Brec research base is made up of recordings coming in particular from video surveillance cameras and on which individuals appear, and more specifically from cameras located in the environment of a place of interest, such as the scene of the crime. Pre-processing of the recordings is therefore preferably carried out in order to extract the video sequences of interest, that is to say on which at least one individual appears.

[0097] At step E3 a filter linked to the class characterization metadata: date is for example applied in order to concentrate the search from, or around the date of interest, such as the actual date of the crime, which would have been entered as identification data.

[0098] Respectively, the Bref reference database is made up of records, in the form of annotated image files for example, which contain individuals known to the investigation services, even though these individuals are not necessarily identified but for example at this stage only associated with a given case.

[0099] In comparison step E4, a biometric comparison is performed.

[0100] Once the comparison step E4 is carried out, the similarity scores are modified in step E5 using said confidence indices.

[0101] As illustrated previously, in this context it is also possible to use referencing metadata, so as to evaluate for a class of characterization metadata the mathematical distance between the value of the characterization metadata and the value of the referencing metadata of the same class, for example here for the size, then to adjust the similarity score at the EA step. Other classes of metadata such as spoken language are also usable especially since the recordings for the Brec research database come from videos, and whose soundtracks are therefore analyzable so that spoken languages ​​can sometimes be identified, the language having also been entered in the Bref reference database by annotation of other investigators for example.

[0102] Figure 3 illustrates the architecture of the reference image ordering system S according to one embodiment of the invention, in the overall context of a decision support system.

[0103] The MAref means for acquiring reference images are represented, such as the camera(s) used in the baggage storage locations without labels; these MAref means for acquiring reference images may already be capable of incorporating referencing metadata into the image file, such as the acquisition date, also called capture date, of the reference Iref image.

[0104] Also represented are the MArec means for acquiring search images, such as the camera(s) used in the baggage check-in locations; these MArec means for acquiring search images may already be capable of incorporating characterization metadata into the image file, such as the date of acquisition, also known as capture, of the Irec search image.

[0105] The reference image scheduling system S includes:

[0106] - an MArec association module which associates with each Irec search image of a Brec search base at least one characterization metadata belonging to a class;

[0107] - a possible (if the referencing metadata are used) MAref association module which associates with each reference Iref image of a reference Bref base at least one referencing metadata (of the same class as at least one characterization metadata or at least one identification data);

[0108] - a DC network communication device (here shown central for reasons of clarity) through which the various information passes, the communications being represented here in particular by the various arrows;

[0109] - a processing device DT that executes computer-readable instructions that, when executed by the processing device, cause the processing device to:

[0110] - receive, by means of the network communication device, a search request for a physical element sought including as search criterion an identification data item for said physical element sought belonging to the same class as the at least one characterization metadata item;

[0111] - filtering the search base according to the identification data of said physical element sought by comparison with the characterization metadata of the same class so as to determine images to be searched for associated with said query; if the filtering of the search base determines at least two images to be searched for associated with said query, said images to be searched for not all corresponding to the same physical element;

[0112] - determine for the class of identification data a confidence index characterizing the link between said physical element sought and said identification data;

[0113] - comparing each reference image of all or part of the reference base to said at least two images to be searched by means of an image comparison algorithm so as to generate a similarity score between said reference image and each of said at least two images to be searched;

[0114] - modify the similarity score generated based on the confidence index of the common class of the identification data and the characterization metadata specific to the Iref image to be searched;

[0115] - order all or part of said reference Iref images by decreasing modified score.

[0116] The ranking of the results is then displayed on a screen connected to the DT processing device or transmitted to a third-party computing device (for example a mobile communication terminal) via the network communication device.

[0117] The DT processing device could be represented containing the MAref, MArec association modules, but it was chosen here to separate them because the respective EA and E1 association steps can be carried out, for example, by an acquisition module which would contain the acquisition means or by a remote server.

[0118] In the case of biometric data, as used in the context of paging, the data is encrypted as it passes through the network communication device.

[0119] Figure 4 is a schematic block diagram of a scheduling system S comprising an information processing device DT for implementing one or more embodiments of the invention.

[0120] This device typically corresponds to the DT processing device in Figure 3. The DT information processing device can be centralized or distributed. The DT information processing device comprises, for example, one or more peripherals such as a microcomputer, a workstation or a remote server on a telecommunications network. The DT processing device comprises a communication bus connected to:

[0121] - a central processing unit 601, such as a microprocessor, denoted CPU;

[0122] - a random access memory 602, denoted RAM, for storing the executable code of the method for implementing the invention as well as the registers adapted to record the calibrations necessary for implementing the method according to embodiments of the invention; the memory capacity of the device can be supplemented by an optional RAM memory connected to an expansion port, for example;

[0123] - a read-only memory 603, denoted ROM, for storing computer programs for implementing the embodiments of the invention;

[0124] - a network interface 604, denoted NET, is normally connected to a communication network on which digital data to be processed are transmitted or received. The network interface 604 may be a single network interface, or composed of a set of different network interfaces (e.g. wired and wireless interfaces or different types of wired or wireless interfaces). Data packets are sent over the network interface for transmission or are read from the network interface for reception under the control of the software application running in the processor 601;

[0125] - a user interface 605 for receiving input from a user or for displaying information to a user;

[0126] - a storage device 606 noted HD capable of storing the Bred and Brec bases, as well as the confidence indices;

[0127] - an input / output module 607 for receiving / sending data from / to external devices such as hard disk, removable storage media or others.

[0128] The executable code may be stored in a non-volatile memory 603, for example a flash memory or a read-only memory, on the storage device 606 or on a removable digital medium such as for example a disk. According to a variant, the executable code of the programs may be received by means of a communication network, via the network interface 604, in order to be stored in one of the storage means of the communication device 600, such as the storage device 606, before being executed.

[0129] The central processing unit 601 is adapted to control and direct the execution of the instructions or portions of software code of the program or programs according to one of the embodiments of the invention, instructions which are stored in one of the aforementioned storage means. After power-up, the CPU 601 is capable of executing instructions from the main RAM memory 602, relating to a software application. Such software, when executed by the processor 601, causes the execution of the described methods.

[0130] The execution of the program may in particular take place on a remote server and its output is then transferred via the network interface 604 to the local GUI user interface 605 for display.

[0131] In this embodiment, the apparatus is a programmable apparatus that uses software to implement the invention. However, as a subsidiary point of view, it should be noted that at least some of the steps of the present invention may be implemented in hardware (for example, in the form of a specific integrated circuit or ASIC (application-specific integrated circuit) or in the form of a programmable logic component or FPGA (field-programmable gate array), for example, on-the-fly calculation operations of certain referencing metadata could be carried out within the baggage drop-off kiosks or image acquisition arches arranged downstream of the check-in in the baggage sorting systems.The invention makes it possible to identify among the Brec search database the best starting points that are the images to be searched and then to classify the results for the adjudication agent or the investigator according to their confidence index, and not only according to the score initially assigned by the image comparison algorithm. The risk linked to exclusion is therefore reduced and the confidence index makes it possible to put the candidates into perspective according to the degree of discrimination of the criterion which is at the origin of the retention of the candidate in question, which improves the efficiency of the ordering process because the order presented will already take into consideration several criteria as well as their value as a criterion.

Claims

Claims 1. Method (P) for ordering reference images (Iref) recorded in a reference base (Bref), said method being implemented by computer and comprising the following steps: - association (E1) with each search image (Irec) of at least one characterization metadata belonging to a class, said search images (Irec) and characterization metadata forming a search base (Brec), - receive (E2) a search request (REQ) for a physical element sought comprising as search criterion an identification data item for said physical element sought belonging to the same class as the at least one characterization metadata item, said class of the identification data item being associated with a confidence index characterizing the link between said physical element sought and said identification data item; - filter (E3) the search base (Brec) according to the identification data of said physical element sought by comparison with the characterization metadata of the same class so as to determine images to be searched for associated with said query; if (COND) the filtering of the search base (Brec) determines at least two images to be searched for associated with said query, said images to be searched not all corresponding to the same physical element: - comparing (E4) each reference image of all or part of the reference base to said at least two images to be searched for by means of an image comparison algorithm so as to generate for said reference image a similarity score between said reference image and each of said at least two images to be searched for; - modify (E5) the similarity score generated based on the confidence index of the said same class, common to the identification data and to the characterization metadata specific to the image to be searched for, in particular by weighting; - order (E6) all or part of said reference images by decreasing modified similarity score.

2. Scheduling method according to the preceding claim, characterized in that the confidence index of a class depends on said class and on a relative quantity of physical elements that said class covers, relative to another class.

3. Scheduling method according to any one of the preceding claims, characterized in that at least one reference metadata is associated with each recorded reference image (Iref), including in particular a reference metadata of the same class as the identification data of said physical element sought.

4. Scheduling method according to the preceding claim, characterized in that it comprises a step (EPF) of pre-filtering the reference base (Bref) by comparing the identification data of said physical element sought with the referencing metadata of the same class so as to select a part of the reference base (Bref) forming a list of candidate reference images (Iref) for comparison with said at least two images to be searched.

5. Scheduling method according to the preceding claim, characterized in that the identification data is a search start date for said physical element sought and the at least one reference image reference metadata (Iref) is a date of acquisition of the reference image, said reference metadata being specific to a context of acquisition of said reference image, the pre-filtering (EPF) of the reference base being a function of said search start date.

6. Scheduling method according to any one of claims 3 to 5, characterized in that the similarity score, specific to the pair of reference image (Iref) and image (Irec) to be searched for, is modified as a function of a difference between the values of the referencing metadata of the reference image (Iref) and the characterization metadata of the image to search for of the same class.

7. Ordering method according to any one of the preceding claims, characterized in that the modification of the similarity score is carried out by weighting, the value of the weighting factor being a function of the confidence index of the common class of the identification data and of the characterization metadata specific to the image to be searched.

8. Scheduling method according to any one of the preceding claims, characterized in that the at least one metadata characterizing or referencing the image (Irec) respectively search or (Iref) reference is extracted by computer analysis of said image (Irec, Iref), in particular by means of neural networks.

9. Scheduling method according to any one of the preceding claims, characterized in that the physical element is a person, the reference images (Iref) being images of referenced individuals and the search images (Irec) being images of unidentified individuals acquired by video surveillance in different locations, the common class of identification data and characterization metadata is for example a date, a location or a size.

10. Scheduling method according to any one of claims 1 to 8, characterized in that the physical element is an object, the object being a piece of luggage, the reference images (Iref) being images of stored luggage and the search images (Irec) being images of enrolled luggage acquired during their enrollment.

11. Scheduling method according to claims 5 and 10 characterized in that the search start date is that of a last registration of said baggage searched or of a last flight made with said baggage.

12. Scheduling method according to any one of the claims 10 to 11 characterized in that the common class of identification data and characterization metadata is a size of the baggage, a weight of the baggage, a color of the baggage, a flexible or rigid nature of the baggage or a data of enrollment of the baggage.

13. Scheduling method according to the preceding claim, characterized in that the metadata characterizing the given class of baggage enrollment is a receipt number, a depositor identifier associated with the baggage, a flight reservation number, a flight number or an enrollment date.

14. System (S) for ordering reference images recorded in a reference base comprising: - an association module (MArec) which associates with each search image (Irec) of a search base (Brec) at least one characterization metadata belonging to a class; - a network communication device (DC); - a processing device (DT) that executes computer-readable instructions that, when executed by the processing device, cause the processing device to: - receiving, by means of the network communication device (DC), a search request (REQ) for a physical element sought comprising as search criterion an identification data item for said physical element sought belonging to the same class as the at least one characterization metadata item, said class of the identification data item being associated with a confidence index characterizing the link between said physical element sought and said identification data item; - filter the search base (Brec) according to the identification data of said physical element sought by comparison with the characterization metadata of the same class so as to determine images to be searched for associated with said query; if the filtering of the search base determines at least two images to be searched for associated with said query, said images to be searched for will not not all corresponding to the same physical element: - comparing each reference image of all or part of the reference base to said at least two images to be searched by means of an image comparison algorithm so as to generate a similarity score between said reference image and each of said at least two images to be searched; - modify the similarity score generated according to the confidence index of said same class, common to the identification data and to the characterization metadata specific to the image to be searched; - order all or part of said reference images by decreasing modified similarity score.

15. Computer program comprising instructions adapted to the implementation of each of the steps of the scheduling method (P) according to any one of claims 1 to 13 when said program is executed on a computer.

Images