System and method to retroactively extract information for noisy data inputs

Abstract

The invention relates to a computer-implemented method comprising performing a first conversion of a digital media data frame into a semantic representation, said first conversion being associated with one or more recognition confidence index, and using as contextual information semantic representations of digital media data frames belonging to a semantic representation set, and if all recognition confidence indexes among said one or more recognition confidence index are above a confidence threshold, adding said semantic representation to said semantic representation set, said semantic representation being associated to the same timestamp than the digital media data frame, wherein said step of performing said first conversion of a digital media data frame is executed, upon the addition of a semantic representation to said semantic representation set, on at least one first digital media data frame associated to a timestamp prior to the timestamp associated to said added semantic representation.

Description

DescriptionTitle: System and Method to Retroactively Extract Information for Noisy Data InputsTechnical Field

[0001] This invention pertains to the field of processing systems, more specifically to methods and systems for information extraction from media data.Background Art

[0002] The extraction of semantic representation from media data involves converting media data into meaningful and actionable insights. This process typically includes transforming media data, such as audio, video, or images, into structured representations that can be easily interpreted and utilized for various applications. For instance, in voice transcription systems (Automatic Speech Recognition or ASR), the goal is to convert spoken words into text that represent the spoken words, while in optical character recognition (OCR), the objective is to translate handwritten or printed text into digital form. Similarly, video activity recognition aims to identify and describe actions or events captured in video footage.

[0003] However, the ability of system to extract semantic information may be limited when the input media data is noisy. Noisy data can arise from various sources, including external factors like background noise, transmission errors, and poor lighting conditions, as well as internal factors such as low speaker volume, heavy accents, or illegible handwriting.

[0004] In the context of speech recognition, noisy data might include audio recordings with background noise, overlapping conversations, or speakers with heavy accents and low volume. For OCR, noisy data could involve scanned documents with smudges, handwritten notes with poor penmanship, or printed text with faded ink. In video activity recognition, noisy data might consist of footage recorded in low light conditions, videos with motion blur, or scenes with multiple overlapping activities.

[0005] These issues complicate the interpretation process, making it difficult to accurately convert raw data into semantic representations.

[0006] Most of the works on denoising the input data, ex. voice I video, focus on elimination of the background noise with use of either hardware (ex. directed microphones) or software solutions (ex. machine learning models) for separating noise from the voice or for enhancing the signal as a whole. But they do not work properly if the data is not understandable at the very source or would require preparing a dedicated model per type of data distortion possible.

[0007] Traditional methods for dealing with noisy data often rely on external contextual information or pretrained models to enhance data quality. For example, voice transcription systems may use lip-reading models to supplement audio data, while OCR systems might employ dictionaries to correct misrecognized words. Despite these efforts, existing approaches have limitations and may not always yield satisfactory results, particularly when the noise originates from the data source itself. These traditional methods have several drawbacks: dependency on External Data: Relying on external contextual information, such as lip-reading models or dictionaries, can be problematic if the external data is not available or is itself noisy.limited Adaptability: Pre-trained models are often designed for specific types of noise and may not generalize well to new or unexpected noise patterns, requiring frequent retraining or fine-tuning, resource usage: the use of external data sources and complex models can be computationally expensive and may require significant storage and processing power. latency Issues: integrating external contextual information can introduce delays in processing, making real-time applications less efficient. incompleteness: these methods may not fully address the root cause of the noise, especially when the noise originates from the data source itself, leading to suboptimal results.

[0008] There is therefore the need for an improved extraction of semantic representation from media data that does not rely on external contextual information.Summary

[0009] This invention improves the situation.

[0010] It is proposed a computer-implemented method comprising: performing a first conversion of a digital media data frame into a semantic representation, said first conversion being associated with one or more recognition confidence index, and using as contextual information semantic representations of digital media data frames belonging to a semantic representation set; and if all recognition confidence indexes among said one or more recognition confidence index are above a confidence threshold, adding said semantic representation to said semantic representation set, said semantic representation being associated to the same timestamp than the digital media data frame; wherein said step of performing said first conversion of a digital media data frame is executed, upon the addition of a semantic representation to said semantic representation set, on at least one first digital media data frame associated to a timestamp prior to the timestamp associated to said added semantic representation.

[0011] The first conversion may in particular be processed on a digital media frame belonging to a plurality of digital media data frames received from one or more digital media recording device, each digital media data frame being associated to a timestamp. In addition, to determine if all recognition confidence indexes among said one or more recognition confidence index are above a confidence threshold, the method may further comprise a step of comparing said one or more recognition confidence index associated with said first conversion to a confidence threshold.

[0012] This way, the semantic representation set may be further enriched, once a new semantic representation is added to this semantic representation set, with an additional semantic representation resulting from the conversion of an earlier digital media data frame which becomes possible once the semantic representation set includes the new semantic representation.

[0013] A semantic representation of a digital media data is a structured and meaningful interpretation of media data that captures the essential information and context in a form that can be easily understood and utilized by both humans and machines. It involves converting unstructured media data, such as audio recordings, video footage, or images of handwritten text, into a format that encapsulates the underlying meaning and relationships within the data.

[0014] For example, in the context of speech recognition, a semantic representation might be the text transcription of spoken words, along with additional metadata such as the time at which the words have been spoken, speaker identity, sentiment, or intent.

[0015] In optical character recognition (OCR), a semantic representation could be the digital text extracted from a scanned document, including information about the document's structure, such as headings, paragraphs, and tables.

[0016] In video activity recognition, a semantic representation might include descriptions of actions or events occurring in the video, along with temporal and spatial relationships between different objects and activities.

[0017] The semantic representations may also be representations such as a summary of the transcribed text, a graph, or an embedding.

[0018] The digital media data frames are converted into semantic representations using contextual information that is provided by the other conversions.

[0019] The conversion is considered as successful, and the resulting semantic representation is added in a semantic representation set, only if the confidence associated with a conversion, expressed by the one or more confidence index, is sufficient.

[0020] When, subsequently, further conversions are successfully performed, the resulting semantic representations are used as an additional context for performing a new conversion of the previously received media data. This additional contextual information may allow disambiguating the conversion, and allow performing the conversion with a sufficient confidence. For example, the conversion of a given data frame that was not successful may become successful when taking into account the additional contextual information.

[0021] Thus, media data that cannot be unambiguously converted to semantic data may be stored until future media data, and semantic data resulting from the conversion of these future media data, allow disambiguating the conversion of these media data. A previous conversion that was already successful can also be improved using the additional contextual information.

[0022] The steps of the method thus allow improving the conversion of media data into semantic representation.

[0023] In another aspect, it is proposed a computing device comprising one or more processing unit configured to execute a method as defined here.

[0024] A processing unit, often referred to as a central processing unit (CPU) or simply a processor, is the primary component of a computer or computing device responsible for executing instructions and performing calculations. It acts as the brain of the system, interpreting and carrying out commands from software applications and the operating system. The processing unit performs a wide range of tasks, including arithmetic operations, logic operations, data manipulation, and control functions.

[0025] In another aspect, it is proposed a computer software comprising instructions to implement at least a part of a method as defined here when the software is executed by a processor.

[0026] In another aspect, it is proposed a computer-readable non-transient recording medium on which a software is registered to implement the method as defined here when the software is executed by a processor.

[0027] The following features, can be optionally implemented, separately or in combination one with the others:

[0028] In an embodiment, the computer-implemented method further comprises performing a second conversion of a digital media data frame into a semantic representation, executed on at least one second digital media data frame upon the reception of said at least one second digital media data frame from one or more digital media recording device, wherein said added semantic representation is the result of said second conversion of said at least one second digital media frame when all recognition confidence indexes among said one or more recognition confidence index associated to said second conversion are above a confidence threshold.

[0029] This allows in particular to enrich further the semantic representation set, when a new semantic representation resulting from the successful conversion of a newly received digital media data frame is added to this semantic representation set, with an additional semantic representation resulting from the conversion of another digital media data frame, which has been received earlier but was not successfully converted yet.

[0030] In an embodiment, digital media data frames are added in a digital media data frames set until they are converted with all recognition confidence indexes among one or more recognition confidence index are above said confidence threshold ; performing said conversion upon the addition of a semantic representation to said semantic representation set is performed on digital media data frames belonging to said digital media data frames set.

[0031] The digital media data frames for which conversions are not successful are thus stored in the digital media data frames set, and, when a new semantic representation is available, a conversion of digital media data frames belonging to the digital media data frame set is performed again. Thus, the digital media data frames are stored until subsequent contextual information allow performing a successful conversion. This allows ensuring that a conversion attempt is performed each time it is possible, for a given digital media data frame, until a successful conversion is performed.

[0032] In an embodiment, the method further comprises, upon adding said semantic representation to said semantic representation set, deleting said digital media data frame.

[0033] The digital media data frame, that often comprises large data such as audio or video data for example, is deleted as early as the digital media data frame can be successfully converted. Thus, the amount of storage needed to store the digital media data frames is reduced.

[0034] In an embodiment, said first and / or second conversion further uses as contextual information digital media data frames associated to semantic representations belonging to said semantic representation set.

[0035] A digital media data frame associated to a semantic representation can provide further contextual information useful for performing the conversion. For example, if the digital media data frames are audio frames, and the semantic representations ASR transcriptions of audio frames, the ASR transcriptions alone can be a useful context for the conversion, but the corresponding audio frames can further improve the conversion, for example by comparing the audio signals.

[0036] In an embodiment, a digital media data frame associated with a semantic representation in the semantic representation set is deleted when all the conversions performed upon the addition of said semantic representation in the semantic representation set are performed.

[0037] When a new semantic representation is added to the semantic representation set, it provides additional context for converting the digital media data frames that were not previously successfully converted. When previous digital media data frames are still present in the digital media data frame set at this stage, one ormore conversions of these previous media data may be performed taking the new semantic representation as additional context.

[0038] The digital media data frame associated to the new semantic representation is also used as additional context for improving this one or more conversion. However, the digital media data frame associated to the new semantic representation is deleted immediately after the one or more conversions performed upon the addition of the new semantic representation are performed and do not require memory storage after.

[0039] Thus, the one or more conversions performed upon the addition of the new semantic representation are improved, while the memory storage requirements are limited.

[0040] These additional features therefore limit storage requirements, while preserving the efficiency improvements provided by the use of digital media data frames associated with semantic representation as contextual information.

[0041] In an embodiment, said conversion of the digital media data frame takes as contextual information semantic representations and digital media data frames having a timestamp whose time difference to the timestamp of the digital media data frame below a duration threshold ; digital media data frames associated to semantic representations belonging to said semantic representation set are deleted, when the time duration between their timestamps and the current time is above said duration threshold.

[0042] Taking as contextual information semantic representations having a timestamp whose time difference to the timestamp of the digital media data frame below a duration threshold allows using as contextual information semantic representations that corresponds to digital media data frames that are temporally close to the digital media data frame that is being converted, thereby resulting into an accurate context.

[0043] In embodiments where the digital media data frames themselves are used as contextual information for the conversion, deleting digital media data frames when the time difference between their timestamp and the current time is above the duration threshold allow removing the digital media data frames that will not be anymore used as contextual information for the conversion of future digital media data frames.

[0044] Therefore, memory storage requirements are reduced.

[0045] In an embodiment, using as contextual information digital media data frames associated to semantic representations belonging to said semantic representation set comprises comparing said digital media data frame to said digital media data frames associated to semantic representations.

[0046] A comparison of a digital media data frames to convert to a digital media data frame that has been successfully converted can improve the conversion.

[0047] For example, if the data frame to convert is noisy and do not allow to detect a word uttered in the data frame, a direct comparison to a similar, but cleaner audio data frame associated with a successful conversion allow determining that the semantic representation is the same than the one of the cleaner audio data frame.

[0048] In an embodiment, using as contextual information semantic representations belonging to said semantic representation set comprises: obtaining a plurality of candidate semantic representations, each of said candidate semantic representation being associated with a recognition confidence index below said confidence threshold; if one of said plurality of candidate semantic representations is present in said sematic representation set, adding said one of said plurality of candidate semantic representations to said semantic representation set.

[0049] When the media data do not allow a disambiguation of the data into a semantic representation, a plurality of candidate semantic representation may be identified with low confidence score. If one of the candidate semantic representations is already present in the semantic representation set, a stronger confidence that this is the right representation can be obtained, and the conversion can thus be disambiguated.

[0050] These features therefore improve the extraction of a semantic representation from media data.

[0051] In an embodiment, using as contextual information semantic representations belonging to said semantic representation set comprises: calculating a similarity between said semantic representation and said semantic representations belonging to said semantic representation set; if said recognition confidence index is below said confidence threshold, and a similarity between said semantic representation and another semantic representation belonging to said semantic representation set is above a similarity threshold, determining said semantic representation to be the another semantic representation with a recognition confidence higher than the confidence threshold.

[0052] Similarity between two semantic representations refers to a measure of how closely related or similar the two representations are in terms of their meaning, structure, or content. This similarity can be quantified using various metrics and techniques, depending on the type of data and the specific application. Common methods to define and calculate similarity between semantic representations include for example : cosine similarity between two vectors. It calculates the cosine of the angle between two vectors in a multi-dimensional space. In the context of semantic representations, these vectors can be word embeddings, sentence embeddings, or any other form of vectorized representation of the data ;Jaccard similarity that measures the similarity between two sets by comparing the size of their intersection to the size of their union. This metric is particularly useful for comparing sets of words, phrases, or other discrete elements ;Euclidian distance that measures the straight-line distance between two points in a multi-dimensional space. In the context of semantic representations, these points can be vectorized representations of the data such as for example words or sentence embeddings.

[0053] If a semantic representation has a low confidence, but is similar to another semantic representation that has been converted with high confidence, it can be inferred that the semantic representation should in fact be this other semantic representation.

[0054] The conversion to semantic representations is thus improved.

[0055] In an embodiment, said one or more digital media recording device comprises one or more microphone ; said digital media data frames are audio frames ; each semantic representation comprises one or more of a text transcription of an audio frame, a summary of an audio frame and an embedding of a text transcription of an audio frame; performing a conversion of a digital media data frame into a semantic representation consists in performing a speech to text conversion of an audio frame into one or more of a text transcription of the audio frame, and a summary of the audio frame.

[0056] This allows performing a speech to text transcription that is more efficient than prior art solutions, especially in case of noisy inputs, because when a noisy audio frame is received, it can be stored to be subsequently disambiguated using future audio frames.

[0057] In an embodiment, said one or more digital media recording device comprises one or more camera ; said digital media data frames are video frames ; each semantic representation comprises one or more of a text description of a video frame, a summary of a text description of a video frame and an embedding of a text description of a video frame ; performing a conversion of a digital media data frame into a semantic representation consists in performing an image semantic description of a video frame into one or more of a text description of a video frame, and a summary of a text description of a video frame.

[0058] This allows performing a video transcription that is more efficient than prior art solutions, especially in case of noisy or low quality inputs, because when a noisy or low quality (e.g low light, blurry...) video frame is received, it can be stored to be subsequently disambiguated using future video frames.

[0059] In an embodiment, said one or more digital media recording device comprises one or more image digital media recording device configured to generate a plurality of image frames ; said digital media data frames are image frames ; each semantic representation comprises one or more of an optical character recognition of an image frame, a summary of an optical character recognition of an image frame, and a summary of an optical character recognition of an image frame; performing a conversion of a digital media data frame into a semantic representation comprises performing an optical character recognition of an image frame.

[0060] This allows performing an OCR that is more efficient than prior art solutions, especially in cases where a handwriting is difficult to recognize, because when characters are not recognized with a sufficient confidence, an image frame can be stored to be subsequently disambiguated using future image frames.Brief Description of Drawings

[0061] Other features, details and advantages will be shown in the following detailed description and on the figures, on which:Fig. 1a

[0062] [Fig. 1 a] is an Automatic Speech Recognition (ASR) system in which the present invention can be implemented according to an embodiment.Fig. 1b

[0063] [Fig. 1 b] is an Optical Character Recognition (OCR) system in which the present invention can be implemented according to an embodiment.Fig. 1c

[0064] [Fig. 1 c] is a video activity recognition system in which the present invention can be implemented according to an embodiment.Fig. 2

[0065] [Fig. 2] is a computing device according to an embodiment of the present invention.Fig. 3

[0066] [Fig. 3] is an example of computing modules implemented by a computing device according to an embodiment of the present invention.Fig. 4

[0067] [Fig. 4] is an example of a computing-implemented method according to an embodiment of the present invention.Description of Embodiments

[0068] It is now referred to figure 1 a.

[0069] Figure 1a is an Automatic Speech Recognition (ASR) system in which the present invention can be implemented according to an embodiment.

[0070] In the system represented in figure 1 a, a microphone Mid a captures the speech uttered by one or more speakers, for example the speaker Spkl a represented in figure 1 a.

[0071] The audio signals captured by the microphone Mid a are then transmitted to a computing device Compl a that processes the signals in order to convert them into a semantic representation Semi a. In practice, the semantic representation Semi a may be a textual representation of the words uttered by the one or more speakers, that can be accompanied by metadata. The metadata may for example comprise one or more a timestamp at which the words have been uttered, a sound level of utterance, a voice intent, etc.

[0072] According to various embodiments of the invention, the computing device Compl a may belong to various types of computing devices. For example, the computing device Compl a may be a server, a personal computer, a tablet, a smartphone, etc. The microphone Mid a may be either an internal microphone of the computing device Compl a (for example an internal microphone of a smartphone or a tablet), a microphone in a wired connection with the computing device Compl a (for example, a microphone of a headset connected to the computing device Compl a through a wire), or a microphone in a wireless connection with the computing device Compl a (for example, a microphone worn by the user that transmits the audio signal to a distant server). The audio signals may be sent in many different ways depending on the link between the microphone Mid a and the computing device Compl a. In particular, raw or compressed audio signals may be sent, in any format that can be interpreted by the computing device Compl a.

[0073] As explained above, all or parts of the audio signals may be noisy. In such case, the extraction of the semantic representation Semi a may be incomplete. For example, some words may be missing because they are not properly understood.

[0074] The present invention may be implemented by the computing device Compl a in order to improve the extraction of the semantic representation of the parts of the audio signals that have not be properly understood. To this effect, as will be demonstrated below, the computing device Compl a can make use of the parts of the audio signals, possibly received after noisy parts, that have been properly understood to disambiguate the extraction of semantic representation from the noisy parts of the audio signals.

[0075] Figure 1b is an Optical Character Recognition (OCR) system in which the present invention can be implemented according to an embodiment.

[0076] In the system represented in figure 1 b, a scanner Scant b scans a plurality of sheets of the manual writing performed by one or more users

[0077] The images signals captured by the scanner Scant b are then transmitted to a computing device Compl b that processes these signals in order to convert them into a semantic representation Semi b. In practice, the semantic representation Semi b may be a textual representation of the manual script from theone or more writers, that can be accompanied by metadata. The metadata may for example comprise a page index, position of a word in a page, color of a ink, etc.

[0078] According to various embodiments of the present invention, the computing device Compl b may belong to various types of computing devices. For example, the computing device Compl b may be a server, a personal computer, a tablet, a smartphone, etc.

[0079] The scanner Scant b may be either an internal scanner of the computing device Compl b (for example if the OCR is performed in the scanner itself), a scanner in a wired connection with the computing device Compl b (for example, a scanner connected to the computing device Compl b through a USB wire), or a scanner in a wireless connection with the computing device Compl b (for example, a scanner that transmits the scans to a computing device in an Ethernet network or to a distant server). The scans may be sent in many different ways depending on the link between the scanner Scant b and the computing device Compl b. In particular, raw or compressed images may be sent, in any format that can be interpreted by the computing device Compl b.

[0080] As explained above, all or parts of the scans may be noisy, for example when some letters or words are not properly written and cannot be transcribed. In such case, the extraction of the semantic representation Semi b may be incomplete. For example, some words may be missing because they are not properly understood.

[0081] The present invention may be implemented by the computing device Compl b in order to improve the extraction of the semantic representation of the parts of scans that have not be properly understood. To this effect, as will be demonstrated below, the computing device Compl b can make use of the parts of the scans, possibly received after noisy parts, that have been properly understood to disambiguate the extraction of semantic representation from the noisy parts of the scans.

[0082] Although the figure 1 b represents an example of OCR using a scan, the invention could be applied to OCRs applied to other kinds of image digital media recording devices than scanners. For example, an OCR can be performed based upon a video recording of a person that is handwriting. In such an example, future frames of the video recording can be used to disambiguate the OCR of previous frames.

[0083] Figure 1c is a video activity recognition system in which the present invention can be implemented according to an embodiment.

[0084] In the system represented in figure 1 c, a camera Caml c records the movements Movi e performed by one or more users

[0085] The video signals captured by the camera Caml c are then transmitted to a computing device Compl c that processes these video signals in order to convert them into a semantic representation Semi c. In practice, the semantic representation Seml c may be a textual representation that describes the activity performed by the users, that can be accompanied by metadata. The metadata may for example comprise a timestamp indicating when the activity has been detected, position of the activity, additional descriptors of the activity (such as for example speed or intensity), etc.

[0086] According to various embodiments of the present invention, the computing device Compl c may belong to various types of computing devices. For example, the computing device Compl c may be a server, a personal computer, a tablet, a smartphone, etc.

[0087] The camera Caml c may be either an internal camera of the computing device Compl c (for example an internal camera of a smartphone or tablet), a camera in a wired connection with the computing device Compl c (for example, a webcam connected to the computing device Compl c through a USB wire), or a camera in a wireless connection with the computing device Compl c (for example, a camera that transmits the videos to a computing device in an Ethernet network or to a distant server). The videos may be sent in many different ways depending on the link between the camera Caml c and the computing device Compl c. In particular, raw or compressed videos may be sent, in any format that can be interpreted by the computing device Compl c.

[0088] As explained above, all or parts of the videos may be noisy, for example when the light is low or movement masked by an object between the camera and the person that performs the movement. In such case, the extraction of the semantic representation Seml c may be incomplete. For example, some activity descriptions may be missing because the activity has not be properly understood.

[0089] The invention may be implemented by the computing device compl c in order to improve the extraction of the semantic representation of the parts of videos that have not be properly understood. To this effect, as will be demonstrated below, the computing device Compl c can make use of the parts of the videos, possibly received after noisy parts, that have be properly understood to disambiguate the extraction of semantic representation from the noisy parts of the video.

[0090] It is now referred to figure 2.

[0091] Figure 2 is a computing device according to an embodiment of the present invention.

[0092] Figure 2 illustrates one embodiment of a computing device Comp2. The computing device Comp2 receives some input data from one or more digital media recording device Sens2 and includes a processing unit Proc2 and a memory Mem2.

[0093] The one or more digital media recording device Sens2 transmits input data to the processing unit Proc2, which processes these data and interacts with the memory Mem2 to store or retrieve information. In particular, the computing device Comp2 may extract semantic representations from media data acquired from the digital media recording device Sens2.

[0094] The digital media recording device Sens2 serves as the data acquisition component for the computing device Comp2. Digital media recording device Sens2 captures environmental data or other forms of measurable input, which may include audio, image visual, or other sensory data. The digital media recording device Sens2 then transmits this data to the processing unit Proc2 for further processing.

[0095] According to various embodiments of the invention, the digital media recording device Sens2 may be either an internal digital media recording device of the computing device Comp2 or an external digital media recording device connected to the computing device Comp2. For example, the digital media recording device Sens2 could be an internal microphone or camera integrated into a smartphone or tablet, or it could be an external device such as a USB-connected microphone, a webcam, or a digital media recording device transmitting data wirelessly to the computing device Comp2.

[0096] The processing unit Proc2 processes the data received from the digital media recording device Sens2. This module may perform a variety of computational tasks, in order to extract semantic representations from the media data. The processing unit Proc2 may also determine the need for data storage or retrieval in conjunction with the memory Mem2.

[0097] The memory Mem2 provides storage capabilities for the computing device Comp2. The memory Mem2 stores processed data, instructions for the processing unit Proc2, and other relevant information. The memory Mem2 may include volatile or non-volatile storage devices, such as RAM, ROM, hard drives, or solid-state drives.

[0098] In a number of embodiments of the present invention, the computing device may implement the computing moules represented in figure 3 and / or the steps of the method P4 represented in figure 4.

[0099] It is now referred to figure 3.

[0100] Figure 3 is an example of computing modules implemented by a computing device according to an embodiment.

[0101] This figure illustrates the various functional modules that work together to achieve the retroactive extraction of information from noisy data inputs. The modules represented in the figure 3 are however provided by non-limitative example only of processing modules that can be implemented according to various embodiments of the invention.

[0102] In the example of figure 3, raw media data is received from one or more digital media recording device, then processed to extract semantic information from the raw media data. The term “raw media data” here designates the data received from the digital media recording device. In practice, the raw media data may be received for example in a raw data format (for example, a bitmap for images, or wav for audio), or in a compressed media data format (for example, jpeg for images or mp3 for audio data). The raw media data may be associated to metadata, for example timestamps or a descriptor of the digital media recording device that captured the data.

[0103] The computing modules of figure 3 notably rely on two storage modules: a raw media data input storage module RawSt3 that stores the raw data received from the one or more digital media recording device. For example, in case of an ASR application, the raw media data input storage RawSt3 may store audio frames associated with timestamps ; a knowledge database Knw3 that stores a semantic representation set comprising the semantic representations extracted from the raw data. For example, in case of an ASR application, the knowledge database Knw3 may store text transcriptions of the speeches heard in audio frames associated with timestamps.

[0104] The modules represented in figure 3 are a non-limitative examples of the modules that may be implemented by a computing device such as the device Comp2. These modules comprise: a preprocessing module Preproc3 responsible for: receiving the raw media data, chunking it for example according to a defined strategy (fixed I from pause to pause, overlapping I not overlapping) and putting it into an ordered list with a timestamp assigned to the raw media data ; a writing module Writ3 responsible for writing into a raw media data input storage RawSt3 (also designated as digital media data frame set) ; a selection module Sel3 responsible for retrieving one of the raw media data chunks in the raw media data storage RawSt3: for example the last one (with the biggest timestamp) upon its arrival in the list or all chunks one by one in a recency order (bigger timestamps before smaller) upon acquiring new semantic representations ;a source recognition module Rec3 that depends on the raw media data type and destination data type (ex. automatic speech recognition I ASR module for transforming voice into text), that takes as input a raw digital media data frames and outputs one or more semantic representation of the raw digital media data frames along with one or more recognition confidence index. For example, in the case of an ASR application, the source recognition module Rec3 can take as input an audio frame and output a text transcription of the audio frame along with one or more recognition confidence index of the transcription. According to various embodiments, the one or more recognition confidence index may be a single value and correspond to the whole audio frame or be a vector of values corresponding to multiple parts of I moments in a given frame. For example, a transcription of an audio frame into text may be accompanied by a vector providing a recognition confidence index for each word ; a knowledge extraction decision module Dec3, which receives the source recognition module Rec3 outputs, that is semantic representations associated with one or more recognition confidence index, and transmit these data into a knowledge extraction module Ext3 if the previously discussed one or more recognition confidence index is above a confidence threshold for the given frame, or the parts of the frame for which the confidence level is above the threshold ; a knowledge extraction module Ext3, which receives the semantic representation when the decision is positive and extracts useful, actionable knowledge (ex. only informative words from text) in a complete or in an aggregated form (ex. “The thing about Paul, you know, he’s my best Friend” -> “Paul is John’s best friend” or (Paul, best-friend-of, John)). The knowledge extraction module Ext3 thus extracts a semantic representation that is not necessarily a raw semantic representation and can be integrated in the knowledge database Knw3, also designated as “semantic representations set” ; a postprocessing module PstProc3 which may perform integration work towards producing a complete and coherent output, i.e. sending into the output interface the transformed form of the data with an original timestamp of the corresponding raw media data (ex. two sentences may be transcribed in an inverse order if the first one needed more processing, so the second sentence is outputted before the first and the timestamps will enable the reconstruction of the data’s order: (“and I’ve been working like a dog” ; t2) -> (“it’s been a hard day’s night ; t1 )) ; a removal module Rem3 for removing unnecessary entries in the raw media data input storage RawSt3, for example on the decision that a digital media data frame has been well transformed or upon initiative to free space - starting with the oldest entries. For example digital media data frames that are too old for being useful for future disambiguation may be removed ; an integration module I nt3 for integrating the extracted knowledge in the knowledge database Knw3, that is the aggregated semantic representations from initial data, with the transformed and aggregated data storage - this may involve simply adding new data, updating the previous information with more recent one or both ; a knowledge database retrieval module Ret3 which searches for the most useful pieces of information for the task of source data recognition.

[0105] It is now referred to figure 4

[0106] Figure 4 is an example of a computing-implemented method according to an embodiment of the present invention.

[0107] The method P4 comprises a first step S41 of receiving a plurality of digital media data frames from one or more digital media recording devices.

[0108] Each digital media data frame is associated with a timestamp. The digital media recording devices can include microphones, cameras, or image digital media recording devices, and the data frames can be audio frames, video frames, or image frames, respectively. The one or more digital media recording devices may for example be the digital media recording devices Sens2, Mid a, Scriptt b or Card c represented in figures 1 a to 1c and 2.

[0109] In general, the objective of the method P4 is to convert the received digital media data frames into semantic representations, and to add the semantic representations into a semantic representation set.

[0110] To this effect, the method P4 further comprises a step S42 of performing a conversion of a digital media data frame into a semantic representation, said conversion being associated with one or more recognition confidence index.

[0111] The recognition confidence index is an index defining to which extent the detection can be considered as reliable. For example, the recognition confidence index can be a value in a range [0, 1 ], where 0 represents a very low confidence, and 1 a very high confidence.

[0112] A conversion of a digital media data frame into a semantic representation involves transforming raw media data into a structured and meaningful interpretation that captures the essential information and context represented in the raw media data. This process can vary depending on the type of media data being processed. Below are detailed examples of conversions for different types of digital media data frames.

[0113] In an example of audio frames, the conversion at step S42 can be an Automatic Speech Recognition or ASR, that may comprise :• Speech-to-Text Transcription: The raw audio data is processed using an ASR system to convert spoken words into text. This may involve recognizing phonemes, words, and sentences from the audio signal ;• Metadata Extraction: Additional information such as timestamps, speaker identity, sentiment, or intent can be extracted frame an audio frame and associated with the text transcription. For example, the system might identify the time at which each word was spoken, the speaker's identity, and the emotional tone of the speech.• Confidence Index Calculation: The ASR system generates a confidence index for the transcription, indicating the likelihood that the recognized text accurately represents the spoken words. This confidence index can be a single value for the entire transcription or a vector of values corresponding to individual words or phrases.

[0114] For example, an audio frame containing the sentence "I live in Warsaw" can be converted into one of the following semantic representations: a word-by-word semantic representation “I live in Warsaw”, associated with a single recognition confidence index, for example 0.95 ; a word-by-word semantic representation “I live in Warsaw”, associated with a plurality of recognition confidence indexes, for example a vector comprising one index for each word, for example [0.95, 0.90, 0.80, 0.83] ;a structured semantic representation that captures the essential meaning, such as [living, Warsaw], associated with a single recognition confidence index, for example 0.95 ; a structured semantic representation that captures the essential meaning, such as [living, Warsaw], associated with a plurality of recognition confidence indexes, for example a vector comprising one index for each element of the structured transcription, for example [0.95, 0.90];

[0115] In an example of video frames, the conversion at step S42 can be a Video Activity Recognition that converts the video frames into a textual description of the activities performed, that may comprise :• Textual Description: The detected activities and objects are converted into a textual description that summarizes the content of the video frame. This description can include information about the actions being performed, the objects involved, and their spatial and temporal relationships.• Metadata Extraction: Additional information such as timestamps, location, and activity descriptors (e.g., speed, intensity) can be extracted and associated with the textual description.• Confidence Index Calculation: The system generates a confidence index for the activity recognition, indicating the likelihood that the detected activities and objects accurately represent the video content.

[0116] Example: Converting a video frame showing a person running can be converted into one of the following semantic representations: a textual semantic representation "A person is running" with a single recognition confidence index of 0.90 ; a structured semantic representation that captures the essential meaning, such as [person, running], associated with a single recognition confidence index, for example 0.90 ; a structured semantic representation that captures the essential meaning, such as [person, running], associated with a plurality of recognition confidence indexes, for example a vector comprising one recognition confidence index for each element of the structured transcription, for example [0.85, 0.95] that represent respectively the confidence that a person has been detected, and the confidence that the action is running.

[0117] In an example of image frames, the conversion at step S42 can be an Optical Character Recognition (OCR) that converts images of handwriting into a textual representation of the written text. In this example the conversion may comprise :• Textual Representation: The recognized text is converted into a digital format, preserving the structure and layout of the original text. This can include information about headings, paragraphs, and tables.• Metadata Extraction: Additional information such as page index, position of words, and ink color can be extracted and associated with the textual representation.• Confidence Index Calculation: The OCR system generates a confidence index for the text recognition, indicating the likelihood that the recognized text accurately represents the content of the image. This confidence index can be a single value for the entire text or a vector of values corresponding to individual characters or words.

[0118] For example, converting an image frame containing handwritten text "Hello World my name is Tom" can be converted into one of the following semantic representations:a word-by-word semantic representation "Hello World my name is Tom", associated with a single recognition confidence index, for example 0.85 ; a word-by-word semantic representation “Hello World my name is Tom”, associated with a plurality of recognition confidence indexes, for example a vector comprising one index for each word, for example [0.85, 0.73, 0.82, 0.83 0.91 0.74] ; a structured semantic representation that captures the essential meaning, such as [Hello, name=Tom], associated with a single recognition confidence index, for example 0.85 ; a structured semantic representation that captures the essential meaning, such as [Hello, name=Tom], associated with a plurality of recognition confidence indexes, for example a vector comprising one index for each element of the structured transcription, for example [0.85, 0.74],

[0119] Extracted semantic representations may also comprise non-textual semantic representations that nevertheless provide an understanding about the content of the digital media data frame. Such non-textural semantic representations may for example comprise representations such as graphs or embeddings.

[0120] The conversion at step S42 may also use, as contextual information for the conversion, semantic representations of digital media data frames belonging to the semantic representation set, i.e semantic representations that result from previous conversions with a sufficiently high confidence.

[0121] Using semantic representations of digital media data frames belonging to the semantic representation set as contextual information for the conversion makes use of the already known semantic representations to improve the conversion. Using semantic representations of digital media data frames belonging to the semantic representation set as contextual information for the conversion can be performed in different ways.

[0122] In an embodiment, the semantic representations at the output of the conversion can be compared to the semantic representations in the semantic representation set.

[0123] In particular, the conversion may output a plurality of candidate semantic representations, each associated to a recognition confidence index. For example, two candidate representations (words in this example) may be identified for a noisy part of an audio frame: “small” with a recognition confidence index of 0.55, and “smart” with a recognition confidence index of 0.60. If the confidence threshold for considering that the recognition is successful is 0.80, none of the two candidate words is considered as sufficiently well recognized.

[0124] The two (or more generally the plurality of) candidate representations can then be disambiguated in different ways.

[0125] For example, it can be checked if one of the semantic representations is already present in the semantic representation set. If one of these plurality of candidate semantic representations is already present in the sematic representation set, it can be determined to be a semantic representation with a recognition confidence degree higher than the confidence threshold. In this example, if “small” but not “smart” is already present in the semantic representation set, this candidate semantic representation is added.

[0126] Indeed, it can be inferred that if the word “small” is already present in the semantic representation set, it is likely that it was the word that was actually uttered. Thus, this word can be added in the semantic representation set even though the sole detection of the word did not provide a sufficient recognition confidence index.

[0127] In another example, a similarity index can be calculated between the candidate semantic representations, resulting from the conversion, and the semantic representations belonging to said semantic representation set. For example, a cosine similarity index or Jaccard similarity index can be calculated between a candidate semantic representation and each semantic representation already present in the semantic representation set.

[0128] Then, if the recognition confidence index associated with a candidate semantic representation is below the required confidence threshold, but the similarity index between this candidate semantic representation and another semantic representation belonging to said semantic representation set is above a similarity threshold, then this candidate semantic representation can be determined to be a semantic representation with a confidence degree higher than the confidence threshold.

[0129] For example, if the word “small” is detected with a recognition confidence index of 0.55 (thus lower than the confidence threshold of 0.80), a cosine similarity index can be calculated between an embedding of the word “small” and embeddings of all the words present in the semantic representations set. If the similarity index with another word already present in the semantic representations set is above a similarity threshold (i.e if the word “small” is considered as being very similar to the another word), it can be inferred that the word was actually the another word already present in the semantic representation set, with a sufficiently high degree of confidence. The confidence threshold can be adapted to the different models according to various embodiments. For example, cosine similarity usually goes from -1 to 1 (wherein 1 means that two expressions have same meaning, 0 means that they have nothing in common (the two expressions are then considered as completely unrelated), and -1 means that the have an opposite meaning. Thus thresholds fairly close to 1 such as 0.8 or 0.9 usually form good thresholds. Of course, such a threshold can be parameter according to various models and embodiments.

[0130] In a number of embodiments, digital media data frames can also be stored and associated to semantic representations. Then, the conversion at step S42 may further use as contextual information digital media data frames associated to semantic representations belonging to said semantic representation set (e.g audio frames in case of an ASR, images in case of an OCR, video frames in case of an automatic gesture recognition, etc.).

[0131] The stored digital media data frames associated to semantic representations belonging to said semantic representation set may be used in different ways.

[0132] In an embodiment, the digital media data frame to convert can be compared to the stored digital media data frames associated to semantic representations already belonging to said semantic representation set, for example to estimate a similarity index between the digital media data frame to convert, and each of the digital media data frames associated to semantic representations belonging to said semantic representation set.

[0133] A non-limitative list of examples of possible comparisons is provided below:For audio frames, the comparison can be performed using techniques such as: o Dynamic Time Warping (DTW): DTW is a technique that measures the similarity between two temporal sequences which may vary in speed. It can be used to align and compare audio frames to find the best match. o Mel-Frequency Cepstral Coefficients (MFCC): MFCCs are coefficients that represent the short-term power spectrum of a sound. By extracting MFCCs from both the digital mediadata frame to convert and the digital media data frames associated with semantic representations, a similarity measure can be computed. o Cosine Similarity: Convert the audio frames into feature vectors (e.g., using embeddings from a pre-trained audio model) and compute the cosine similarity between the vectors to determine how similar the audio frames are.For video frames, the comparison can be performed using techniques such as: o Frame-by-Frame Comparison: Compare individual frames of the video using image similarity metrics such as Structural Similarity Index (SSIM) or Mean Squared Error (MSE). o Feature Extraction: Extract features from the video frames using convolutional neural networks (CNNs) and compare the feature vectors using cosine similarity or Euclidean distance. o Optical Flow: Use optical flow to compare the motion patterns between video frames. This can help in identifying similar activities or movements.For image frames used in Optical Character Recognition (OCR), the comparison can be performed using techniques such as: o Template Matching: Use template matching to compare the image frame to known templates of characters or words. o Feature Matching: Extract features from the image frames using techniques like Scale- Invariant Feature Transform (SIFT) or Speeded-Up Robust Features (SURF) and match the features between the frames. o Histogram Comparison: Compare the histograms of the image frames to determine similarity. This can be useful for comparing the overall distribution of pixel intensities.

[0134] In all cases, the comparison results can be used to determine the similarity between the digital media data frame to convert and the stored digital media data frames associated with semantic representations. If a high similarity is found, it can be inferred that the semantic representation of the digital media data frame to convert is likely to be the same as the semantic representation of the similar digital media data frame.

[0135] For example, a chosen similarity metric can be computed between the digital media data frame to convert and each stored digital media data frames associated with semantic representations belonging to the semantic representation set. If the similarity metric between the digital media data frame to convert and another digital media data frame associated with a given semantic representation belonging to the semantic representation set is above a threshold, it can be inferred that the digital media data frame to convert and this other digital media data frame actually represent the same thing. Thus the digital media data frame can be converted in the given semantic representation with a high degree of confidence.

[0136] Then, at the output of the step S42 of conversion, the one or more recognition confidence index associated to semantic representations obtained through the conversion are compared to a confidence threshold Ts (step S43).

[0137] The comparison of this one or more recognition confidence index to the confidence threshold provides an indication as to whether the conversion can be considered as correct or not. The confidence threshold can be obtained in different ways. For example, it can be a predefined confidence threshold. The confidence threshold can be for example predefined by a user according to previous observations, or learned during a training phase in order to set the right threshold between successful and unsuccessful conversions.

[0138] At step S44, the comparison is verified, i.e it is verified if all confidence indexes among said one or more recognition confidence indexes are above a confidence threshold or not. If all confidence indexes among said one or more recognition confidence index are above a confidence threshold, it can be inferred that at the output of the conversion the semantic representation accurately represent the content of the digital media data frame. In such case the semantic representation is added in a step S46 to the semantic representation set, and is associated to the same timestamp than the digital media data frame.

[0139] Thus, the semantic representation set comprises only semantic representations associated with a high level of confidence. Therefore, it can be inferred that all the semantic representations in the semantic representation set correspond top elements that were present in the digital media data frame.

[0140] In a first scenario, steps S42 to S44 are performed upon the reception of an input data frame from the one or more digital media recording device, on said input data frame. Stated otherwise, when an input data frame is received, it is converted into a semantic representation and, if the conversion is successful (e.g if all confidence indexes among said one or more recognition confidence index are above the confidence threshold), the semantic representation that results from the conversion is added in the semantic representation set.

[0141] Conversely, when a conversion is performed upon the reception of an input data frame and if not all confidence indexes among said one or more recognition confidence index are above a confidence threshold, it can be inferred that at least one element of the received input digital media data frame has not been properly understood. In such case, the method P4 comprises a step S45 of adding the input digital media data frame in a digital media data frame set.

[0142] Thus, when an input digital media data frame cannot be converted with a sufficiently high degree of confidence, it is added into the digital media data frame set in order to be converted again later, when new semantic representations become available in the semantic representation set.

[0143] In a second scenario, when a semantic representation is added to the semantic representation set (e.g when a digital media data frame is converted with a sufficiently high level of confidence), all or a part of the digital media data frames that are present in the digital media data frames set can be converted, using as contextual information the semantic representations that belongs to the semantic representations set.

[0144] Thus, the digital media data frames in the digital media data frames set, for which the previous conversions were not successful, can be processed again for another conversion, with additional contextual information that may render this other conversion more successful.

[0145] In particular, upon the addition of a semantic representation to said semantic representation set, at least one digital media data frame associated to a timestamp prior to the timestamp associated to said added semantic representation may be converted using the newly added semantic representation as contextual information.

[0146] Therefore, media data that cannot be unambiguously converted to semantic data may be stored until future media data, and semantic data resulting from the conversion of these future media data, allow disambiguating the conversion of these media data. Newly added semantic representations may also be used as additional context for previous successful conversions, in order to even improve conversions that were already successful.

[0147] When a semantic representation is added to the semantic representation set (e.g when a conversion of a digital media data frame is successful), the corresponding digital media data frame (i.e. from which thissemantic representation has been converted from) may be either stored or kept (if already present) in the digital media frames set, or deleted.

[0148] Storing or keeping the digital media data frames allows preserving the possibility of performing an even better conversion when more contextual information is available later.

[0149] Conversely, deleting later the digital media data frame allows reducing the amount of storage when a conversion is performed with an expected level of confidence. This allows a significant reduction of storage, as the digital media data frames often comprise large amount of data, especially in the case of image, audio or video data.

[0150] An option for managing an input digital media data frame consists in: when the conversion of the input digital media data frame is not successful (e.g if one of the one or more recognition confidence indexes associated with the converted digital media data frame is below the confidence threshold), storing the digital media data frame in the digital media data frames set; when the conversion of the input digital media data frame is successful (e.g if all of the one or more recognition confidence indexes associated with the converted digital media data frames are all above the confidence threshold): o adding the semantic representation resulting from the conversion in the semantic representation set ; o upon the addition of said semantic representation in the semantic representation set, performing further conversions of digital media data frames that were previously stored in the digital media data frames set, using as contextual information the newly added semantic representation resulting from the successful conversion, and the input digital media data frame whose conversion was successful ; o once all the further conversions of data frames are performed, deleting the input digital media data frame.

[0151] Therefore, when a successful conversion provides new contextual information for interpreting the digital media data frames in the digital media data frames set, both the semantic representation and the input digital media data frame involved in the successful conversion are used as contextual information for performing the further conversions of the digital media data frames in the digital media data frames set (e.g the digital media data frames for which the conversion was not successful, and / or digital media data frames for which the conversion was successful but may be improved). Thus, all the possible information is taken into account for the further conversions.

[0152] Meanwhile, the input digital media data frame may be deleted immediately after its use as contextual information. Thus, the storage requirements are limited. Therefore, this solution allows in the same time to benefit from high quality conversions and to limit the storage requirements.

[0153] In a number of embodiments of the invention, a conversion of the digital media data frame takes, as contextual information, semantic representations and digital media data frames having a timestamp whose time difference to the timestamp of the digital media data frame below a duration threshold.

[0154] Stated otherwise, only the semantic representations and digital media data frames considered as being sufficiently close temporally are taken into account. It is in some embodiments expected that the semanticrepresentations are correlated, when the time duration between their timestamps and the current time is above said duration threshold correlated only over a limited time range. For example, in a text transcription of an audio stream, the similar semantic representations may be found only during a limited time duration, for example during the recording of a dialog.

[0155] In this context, one possibility for managing the digital media data frames may consist in: when the conversion of an input digital media data frame is not successful (e.g if one of the one or more recognition confidence indexes associated with the conversion is below the confidence threshold), storing the digital media data frame in the digital media data frames set; when the conversion of the input digital media data frame is successful (e.g if all of the one or more recognition confidence indexes associated with the converted digital media data frames are above the confidence threshold): o adding the semantic representation resulting from the conversion in the semantic representation set ; o storing the input digital media data frame ; o deleting the input digital media data frame, when the time duration between their timestamps and the current time is above said duration threshold.

[0156] Stated otherwise, when the conversion of an input data frame is successful and the resulting semantic representation is stored, the input data frame is also stored, but only for a duration equal to the duration threshold. This duration during which the input data frame is stored corresponds to the duration during which the input data frame can be used as contextual information. Once the duration is over, and the input data frame will not be used anymore as contextual information, the input data frame is deleted.

[0157] This way, further conversions can benefit from the input data frame as a contextual information, while limiting the storage requirements because the input data frame will be deleted as early as it will not have a chance to be used again.

[0158] The duration threshold may be calculated in different ways. The duration threshold may for example be defined by: conducting experiments, using application specific requirements, using machine learning models or through human expertise.

[0159] Example of application to an ASR system.

[0160] A concrete example of the invention will now be described.

[0161] In this example, a video recording of a person speaking needs to be transcribed into text but the person is speaking low and with an accent making it hard for the automatic speech recognition (ASR) system to interpret correctly at all times. However, there are moments where the voice is louder and clearer. The recording is streamed and the ASR system tries to interpret it in real time by chunks of limited duration (fixed I from pause to pause, overlapping I not overlapping). The ASR outputs two values: the recognized text (semantic information) and the recognition confidence level.

[0162] If the recognition confidence level is above a given confidence threshold, the text is stored and the recording is deleted. If the recognition confidence level is below the confidence threshold, the recording is stored for later, portions of text being possibly stored then if the recognition confidence level is not a singlevalue but a series of values corresponding to different parts of the recording. Later, when new portions of speech are converted into text, the newly received text and audio frames are used for improving the conversion of the preceding audio portions that were not understood.

[0163] In this example, the following audio frames are received, at 6 successive timestamps tO to t6:I live in Warsaw, in a very s...<inaudible>...t, actuallyAnd you own this flat?- No.Do you like it in there?Well, as I said, it’s tiny, too small for m...<inaudible>...And how big is your family again?

[0164] In this example, the ASR converts the raw audio frames into an aggregated form that captures the essential meaning of the audio frame. Once an audio frame, or part of an audio frame, is successfully converted, the semantic representation is stored in an aggregated form in the semantic representations set. The raw audio that has not been successfully converted is stored in the digital media data frames set. The semantic representation and raw audio are associated with their timestamps, noted tO, t1 , etc.

[0165] At timestamp tO, the following audio is received: “I live in Warsaw, in a very s.. ,<inaudible>...t, actually."

[0166] The following semantic representations and audio frames are then stored as follows:Semantic representations set: [(lives in Warsaw ; tO)]Digital media data frames set: [(in a very s. .. [inaudible]. ..t, actually ; tO)]

[0167] A timestamp t1 , the following audio is received: “And you own this flat?”.

[0168] The following semantic representations and audio frames are then stored as follows:Semantic representations set: [(lives in Warsaw ; tO)]Digital media data frames set: [(in a very s. .. [inaudible]. ..t, actually ; tO)]

[0169] A timestamp t2, the following audio is received: “No”.

[0170] The following semantic representations and audio frames are then stored as follows:Semantic representations set: [(lives in Warsaw ; tO), (lives in a flat ; t1 ), (doesn’t own the flat ; t2)]Digital media data frames set: [(in a very s...[inaudible]. ..t, actually ; tO)]

[0171] A timestamp t4, the following audio is received: “Well, as I said, it’s tiny, too small for m...<inaudible>...”

[0172] At this stage, the detection of the word “small” associated with the flat in the audio at t4 can be used as a contextual information to interpret the audio received at tO [(in a very s... [inaudible]... t, actually ; tO)], which means that the speaker leaves in a small flat.

[0173] The following semantic representations and audio frames are then stored as follows:Semantic representations set: [[(lives in Warsaw ; tO), (lives in a flat ; t1 ), (doesn’t own the flat ; t2), (the flat is small; tO)],Digital media data frames set: [(too small for m...<inaudible>... ; t4)]

[0174] A timestamp t5, the following audio is received: “And how big is your family again?”. At this stage, the reception of this audio mentioning the family allows disambiguating the audio received at t4, which meant that the small was too flat for the family.

[0175] The following semantic representations and audio frames are then stored as follows:Semantic representations set: [[[(lives in Warsaw ; tO), (lives in a flat ; t1 ), (doesn’t own the flat ; t2), (the flat is small; tO), (has family ; t5)],Digital media data frames set: [()]

[0176] At the end of this process, the following complete transcription of the dialog can be completed:I live in Warsaw, in a very small flat, actuallyAnd you own this flat?- No.Do you like it in there?Well, as I said, it’s tiny, too small for my familyAnd how big is your family again?

[0177] This example demonstrates that digital media data frames that are not initially successfully converted into semantic representations can be stored, and then converted successfully later, using additional contextual information created by the conversion of digital media data frames received subsequently.

[0178] This example demonstrates how the invention can be implemented to improve the conversion of digital media data frames into semantic representations.

[0179] This invention is not limited to the method, device and computer program described here, which are only examples. The invention encompasses every alternative that a person skilled in the art would envisage when reading this invention.

[0180] Recitation of aspects of the invention

[0181] The following clauses set forth various aspects of embodiments of the invention:A computer-implemented method comprising: receiving a plurality of digital media data frames from one or more digital media recording device, each digital media data frame being associated to a timestamp ; performing a conversion of a digital media data frame into a semantic representation, said conversion being associated with one or more recognition confidence index, and using as contextual information semantic representations of digital media data frames belonging to a semantic representation set ; comparing said one or more recognition confidence index to a confidence threshold ; and: if all recognition confidence indexes among said one or more recognition confidence index are above a confidence threshold, adding said semantic representation to said semantic representation set, said semantic representation being associated to the same timestamp than the digital media data frame ; wherein said step of performing said conversion of said digital media data frame is executed upon the reception of an input data frame from the one or more digital media recording device, on said input data frame ; and / or upon the addition of a semantic representation to said semantic representation set, on at least one digital media data frame associated to a timestamp prior to the timestamp associated to said added semantic representation.

[0182] In a further arrangement, digital media data frames are added in a digital media data frames set until they are converted with all recognition confidence indexes among one or more recognition confidence index are above said confidence threshold, and performing said conversion upon the addition of a semantic representation to said semantic representation set is performed on digital media data frames belonging to said digital media data frames set.

[0183] In a further arrangement, this method comprises, upon adding said semantic representation to said semantic representation set, deleting said digital media data frame.

[0184] In a further arrangement, said conversion further uses as contextual information digital media data frames associated to semantic representations belonging to said semantic representation set.

[0185] In a further arrangement, a digital media data frame associated with a semantic representation in the semantic representation set is deleted when all the conversions performed upon the addition of said semantic representation in the semantic representation set are performed.

[0186] In a further arrangement, said conversion of the digital media data frame takes as contextual information semantic representations and digital media data frames having a timestamp whose time difference to the timestamp of the digital media data frame below a duration threshold, and digital media data frames associated to semantic representations belonging to said semantic representation set are deleted, when the time duration between their timestamps and the current time is above said duration threshold.

[0187] In a further arrangement, using as contextual information digital media data frames associated to semantic representations belonging to said semantic representation set comprises comparing said digital media data frame to said digital media data frames associated to semantic representations.

[0188] In a further arrangement, using as contextual information semantic representations belonging to said semantic representation set comprises obtaining a plurality of candidate semantic representations, each of said candidate semantic representation being associated with a recognition confidence index below said confidence threshold, and, if one of said plurality of candidate semantic representations is present in said sematic representation set, adding said one of said plurality of candidate semantic representations to said semantic representation set.

[0189] In a further arrangement, using as contextual information semantic representations belonging to said semantic representation set comprises calculating a similarity between said semantic representation and said semantic representations belonging to said semantic representation set, and, if said recognition confidence index is below said confidence threshold, and a similarity between said semantic representation and another semantic representation belonging to said semantic representation set is above a similarity threshold, determining said semantic representation to be the another semantic representation with a recognition confidence higher than the confidence threshold.

[0190] In a further arrangement: said one or more digital media recording device comprises one or more microphone (Mid a) ; said digital media data frames are audio frames ; each semantic representation comprises one or more of a text transcription of an audio frame, a summary of an audio frame and an embedding of a text transcription of an audio frame;performing a conversion of a digital media data frame into a semantic representation consists in performing a speech to text conversion of an audio frame into one or more of a text transcription of the audio frame, and a summary of the audio frame.

[0191] In a further arrangement: said one or more digital media recording device comprises one or more camera (Caml c) ; said digital media data frames are video frames ; each semantic representation comprises one or more of a text description of a video frame, a summary of a text description of a video frame and an embedding of a text description of a video frame ; performing a conversion of a digital media data frame into a semantic representation consists in performing an image semantic description of a video frame into one or more of a text description of a video frame, and a summary of a text description of a video frame.

[0192] In a further arrangement: said one or more digital media recording device comprises one or more image digital media recording device configured to generate a plurality of image frames (Scant b) ; said digital media data frames are image frames ; each semantic representation comprises one or more of an optical character recognition of an image frame, a summary of an optical character recognition of an image frame, and a summary of an optical character recognition of an image frame.

Claims

Claims

1. A computer-implemented method (P4) comprising: performing (S42) a first conversion of a digital media data frame into a semantic representation, said first conversion being associated with one or more recognition confidence index, and using as contextual information semantic representations of digital media data frames belonging to a semantic representation set (Semi a; Semi b; Seml c); and if all recognition confidence indexes among said one or more recognition confidence index are above a confidence threshold (S44), adding (S46) said semantic representation to said semantic representation set, said semantic representation being associated to the same timestamp than the digital media data frame; wherein said step of performing said first conversion of a digital media data frame is executed, upon the addition of a semantic representation to said semantic representation set, on at least one first digital media data frame associated to a timestamp prior to the timestamp associated to said added semantic representation.

2. The computer-implemented method of claim 1 , further comprising : performing a second conversion of a digital media data frame into a semantic representation, executed on at least one second digital media data frame upon the reception of said at least one second digital media data frame from one or more digital media recording device (Sens2; Mid a; Scriptl b; Caml c), wherein said added semantic representation is the result of said second conversion of said at least one second digital media frame when all recognition confidence indexes among said one or more recognition confidence index associated to said second conversion are above a confidence threshold (S44).

3. The computer-implemented method of claim 1 or 2, wherein: digital media data frames are added (S45) in a digital media data frames set until they are converted with all recognition confidence indexes among one or more recognition confidence index are above said confidence threshold ; performing said first conversion upon the addition of a semantic representation to said semantic representation set is performed on digital media data frames belonging to said digital media data frames set.

4. The computer-implemented method of any one of claims 1 to 3, further comprising, upon adding (S46) said semantic representation to said semantic representation set, deleting said digital media data frame.

5. The computer-implemented method of any one of claims 1 to 4, wherein said first and / or second conversion further uses as contextual information digital media data frames associated to semantic representations belonging to said semantic representation set.

6. The computer-implemented method of claim 5, wherein a digital media data frame associated with a semantic representation in the semantic representation set is deleted when all the conversions performed upon the addition of said semantic representation in the semantic representation set are performed.

7. The computer-implemented method of claim 5, wherein: said first and / or second conversion of the digital media data frame takes as contextual information semantic representations and digital media data frames having a timestamp whose time difference to the timestamp of the digital media data frame below a duration threshold ; digital media data frames associated to semantic representations belonging to said semantic representation set are deleted, when the time duration between their timestamps and the current time is above said duration threshold.

8. The computer-implemented of any one of claims 5 to 7, wherein using as contextual information digital media data frames associated to semantic representations belonging to said semantic representation set comprises comparing said digital media data frame to said digital media data frames associated to semantic representations.

9. The computer-implemented of any one of claims 1 to 8, wherein using as contextual information semantic representations belonging to said semantic representation set comprises: obtaining a plurality of candidate semantic representations, each of said candidate semantic representation being associated with a recognition confidence index below said confidence threshold ; if one of said plurality of candidate semantic representations is present in said sematic representation set, adding said one of said plurality of candidate semantic representations to said semantic representation set.

10. The computer-implemented of any one of claims 1 to 8, wherein using as contextual information semantic representations belonging to said semantic representation set comprises: calculating a similarity between said semantic representation and said semantic representations belonging to said semantic representation set; if said recognition confidence index is below said confidence threshold, and a similarity between said semantic representation and another semantic representation belonging to said semantic representation set is above a similarity threshold, determining said semantic representation to be the another semantic representation with a recognition confidence higher than the confidence threshold.

11. The computer-implemented method of any of the preceding claims, wherein: said one or more digital media recording device comprises one or more microphone (Mid a) ; said digital media data frames are audio frames ;each semantic representation comprises one or more of a text transcription of an audio frame, a summary of an audio frame and an embedding of a text transcription of an audio frame; performing a conversion of a digital media data frame into a semantic representation consists in performing a speech to text conversion of an audio frame into one or more of a text transcription of the audio frame, and a summary of the audio frame.

12. The computer-implemented method of any of claims 1 to 10, wherein: said one or more digital media recording device comprises one or more camera (Caml c) ; said digital media data frames are video frames ; each semantic representation comprises one or more of a text description of a video frame, a summary of a text description of a video frame and an embedding of a text description of a video frame ; performing a conversion of a digital media data frame into a semantic representation consists in performing an image semantic description of a video frame into one or more of a text description of a video frame, and a summary of a text description of a video frame.

13. The computer-implemented method of any of claims 1 to 10, wherein: said one or more digital media recording device comprises one or more image digital media recording device configured to generate a plurality of image frames (Scant b) ; said digital media data frames are image frames ; each semantic representation comprises one or more of an optical character recognition of an image frame, a summary of an optical character recognition of an image frame, and a summary of an optical character recognition of an image frame; performing a conversion of a digital media data frame into a semantic representation comprises performing an optical character recognition of an image frame.

14. A computing device (Comp2; Compl a; Compl b; Compl c) comprising one or more processing unit (Proc2) configured to execute a method according to one of claims 1 to 13.

15. Computer software comprising instructions to implement at least a part of a method according to one of claims 1 to 13 when the software is executed by a processor.

Images