Method for managing the communication of an item of vector data

Abstract

The invention relates to a method for managing the communication of an item of vector data (SD), referred to as the item of data to be communicated, from a first device (EQ1) to a second device (EQ2), the method comprising: obtaining an index (I') by applying an indexing function (F) to the item of data to be communicated (SD), wherein the indexing function (F) is such that the closer two items of vector data are, as measured by a distance (D) between them, the closer the indices calculated for these two items of vector data by the function (F) are to one another, wherein the index (I') makes it possible to retrieve, in a data structure (DS), another item of vector data (SD'), referred to as the item of communicated data, wherein the data structure (DS) associates indices with items of vector data, and wherein the index (I') is capable of being transmitted from the first device (EQ1) to the second device (EQ2).

Description

[0001] Description

[0002] Title: Method for managing the communication of vector data

[0003] Technical field

[0004] The technical field is that of telecommunications.

[0005] More specifically, the invention relates to a method for managing the communication of vector data. The communicated vector data represents semantic content.

[0006] Data will be communicated by the method according to the invention from one device to another device. Such data is a particular type of data that is used in the field of artificial intelligence. The capabilities of artificial intelligence, and in particular of generative artificial intelligence (translation of the English generative artificial intelligence), that is to say the branch of artificial intelligence that focuses on the automatic production of media, in particular texts, images or videos, appeared in the eyes of the general public with the launch of Chat-GPT (product of the company Open Al), the first version of which dates from November 2022. Other generative artificial intelligence programs are developed by the companies Google (product Gemini), Anthropic (product Claude) or Mistral Al (product Le Chat).The products mentioned here are conversational agents, but other generative artificial intelligence products can produce images (DALL-E products from the company Open Al, or Midjourney, from the research laboratory of the same name), videos, sounds, but also computer programs or even specifications of chemical molecules. Other artificial intelligence tasks can concern, for example, pattern recognition, decision-making, classification, planning or any other type of activity imitating the capabilities of human intelligence.

[0007] Generative AI programs encode the data they manipulate for processing in a format that we call vector data. We also use the terms semantic data or semantic vector. Semantic content, the meaning expected or interpreted by users of a text, image, or video, is therefore associated with vector data. Such vector data is a high-dimensional vector in the space of real numbers. The typical number of dimensions for vector data ranges from several hundred to several thousand. The MTEB (Massive Text Embedding Benchmark) project conducts performance comparisons between models used by generative AI programs, and as of March 2024, of the ten best-performing models, seven used vectors of dimension 4096 and three of dimension 1024.

[0008] The term most often used in English to talk about vector data manipulated by generative artificial intelligence programs is embedding. However, this term confuses the function that will encode data such as text or an image for further processing, a mathematical function that is an embedding (translation of the English embedding) and the data obtained as a result of the application of this function. To avoid this confusion between a function and its result, we will generally use the term vector data in this document, even if the term embedding is more often found in the literature, in English and French.

[0009] Vector data are therefore characterized by the fact that they represent semantic content in the form of computer data. It is possible to see that semantic content is indeed associated with vector data by looking at the transformations that can be applied to them. A classic example in artificial intelligence considers a data corresponding to the concept of "France" which can be transformed into another data corresponding to the concept of "Paris" and the same transformation, applied to the vector data associated with "Germany", will produce the vector data associated with "Berlin" and the same for data "Spain" - "Madrid", "Italy" - "Rome" etc. This indicates that a semantic content corresponding to a country or a city is indeed associated with the corresponding vector data and that manipulations of the vector data correspond to manipulations of the concepts associated with the vector data.

[0010] Vector data, which initially emerged in the context of generative artificial intelligence, has found uses across the spectrum of artificial intelligence programs.

[0011] As mentioned, the data we are talking about are vectors of real numbers ranging in size from several hundred to several thousand. These several hundred or thousands of real numbers that form a vector data are generally represented for computer processing by a data type called a floating point number (translation from English float) which uses 32 bits to be represented, that is to say 4 bytes. A data that is a vector of dimension 4096 will therefore have an approximate memory size of 16 KB (16 kilobytes, or 16 kilobytes in English, or 16 Kb).

[0012] It can be noted that the number of dimensions of vector data tends to increase over time, and that the best performing models tend to use higher dimensional vectors.

[0013] Some artificial intelligence researchers propose using vectors of complex numbers to represent concepts. Such a proposal, if adopted and the number of dimensions remained constant, would involve doubling the size of the vector data by replacing a 32-bit floating-point real number with two real numbers to represent a complex number.

[0014] Furthermore, we can talk about data communication in several cases. For example, there is communication when computer data is transmitted over a computer network, which is local to several computers, or over a telecommunications network which can be local or long distance, or when data is transferred from one medium to another, that is to say for example copied from a computer file system to a memory medium connected to the computer such as a USB key or an external hard drive, or vice versa. In all these cases of communication, transmission errors can occur. Computer data are classically defined as sequences of bits. When communicating data, one or more bits constituting this data can be modified by the transmission channel. In this case, there is a transmission error, and the data is no longer transmitted in its integrity and is therefore not communicated without error.

[0015] We have just talked about bit sequences to define computer data. However, in general, we should rather talk about symbol sequences. Indeed, bits are symbols that come in pairs (0 and 1) but the transmissions actually carried out at the physical level rarely concern such symbols that come in pairs. The symbols transmitted at the physical level will be, for example, electromagnetic waves to which phase-shift modulations are applied in order to transmit one value among 4, 8 or 16 values. These techniques are used, for example, in mobile telecommunications. Phase-shift modulations are called phase-shift keying in English and we find the acronyms 4-PSK, 8-PSK or 16-PSK to describe phase-shift modulations that allow one value among 4, 8 or 16 values ​​to be transmitted in a single transmission.In fiber optic transmissions, the use of wavelengths also makes it possible to transmit symbols that can take one of many possible values, and not just symbols such as bits that take their values ​​from a pair of possible values. Even though the transmitted symbols themselves are not bits and can take more than two values, there is still a direct link between a transmitted symbol and bits. For example, in the case of 4-PSK modulation, the transmitted symbols can take four values, and these four values ​​correspond respectively to the bit pairs '00', '01', '10', and '11'.

[0016] In this document we will therefore talk about the general concept of a sequence of symbols which can be understood in a more simplified reading as being a sequence of bits.

[0017] These points are well known to those skilled in the art and will not be developed further.

[0018] State of the art

[0019] The upcoming development of artificial intelligence promises a future explosion in the transmission of vector data representing semantic content. Indeed, one can easily imagine interesting applications in which vector data would be communicated from one device that produces it to another device that will use it.

[0020] For example, during a telephone conversation, a speech recognition system coupled with an interpretation system could interpret each sentence spoken by a speaker into vector data. The meaning of the sentence spoken by the speaker would then be associated with the vector data. This data would be communicated to another speaker who would have a reverse interpretation system coupled with a speech synthesis system. This system would allow for automatic translation of the telephone conversation, regardless of the languages ​​used by the speakers. In a conference call with speakers each speaking their own language, the speakers would hear each speaker's interventions translated on the fly into their own language.

[0021] To give another example, an artificial intelligence system can detect events such as an attack or a burglary on a video filmed by a CCTV camera. A user may wish to have such a video surveillance service without the video stream filming their home being transmitted outside. Rather than transmitting the video, an artificial intelligence video surveillance system can produce vector data with semantic content and communicate it. A system trained to detect intrusions will then only need this vector data to detect possible intrusions and the confidentiality of video surveillance users will be preserved.

[0022] In the field of telecommunications, research is beginning to explore the concept of semantic networks that automatically adapt to the needs of applications transmitting data over the network using artificial intelligence techniques. Such semantic networks must rely on general representations of information that lend themselves to automatic adaptations and optimizations. These representations can carry semantic data of various kinds, such as images, text, or even configuration elements of the semantic network itself. A semantic network therefore relies on semantic vector data as its elementary data format.

[0023] These possible application examples demonstrate the general interest in transmitting vector data (often called embeddings in English) in computer and / or telecommunications networks or, more generally, in interconnecting artificial intelligence systems in order to perform complex tasks.

[0024] More generally, vector data is a multimodal representation. It is possible to represent all kinds of data (texts, images, videos, etc.) in the form of vector data, knowing that the semantic contents of these multimodal representations are associated with the vector data. Vector data with semantic content could therefore be used as a universal data format within telecommunications networks instead of the bit sequences used today. Such data would then present a finer granularity of information and more advanced optimization possibilities.

[0025] The interconnection of artificial intelligence applications or the deployment of semantic networks will involve an explosion in the quantity of vector data to be communicated between devices. These vector data communications will correspond to the effective transmission of symbols between devices. The problem to be solved is therefore to communicate vector data from a transmitter to a receiver while preserving the semantic content associated with the vector data to be communicated while saving the number of symbols actually transmitted. The scientific field of semantic communications is interested in the problem of communicating only the minimal content of the data necessary to carry out a task and in particular the content necessary to make artificial intelligence systems communicate with each other.The prevailing approach in this field is to communicate only data relevant to a targeted task in order to limit the memory size of the transmitted elements in terms of bits or bytes transmitted. In the example given above of video surveillance, a pre-transmission processing will seek to detect images (or extracts of images) suspected of representing an intrusion and only these extracts will be communicated, by removing images or semantic data from elements deemed useless to the video surveillance task. For example, color could be removed before transmission in this approach.

[0026] A major drawback of this approach is that the processing performed before transmission depends on the task that is expected at reception. However, this task at reception cannot be known in all cases. Knowing this task at reception also implies that the telecommunications operator (which transmits the data and will often prepare it for transmission) must be informed of the tasks performed by the applications that use its network. However, telecommunications operators are generally agnostic about the applications that use their networks. And it would also be very impractical to develop network optimizations specific to each expected processing task.For these reasons, such an approach could only be applied in the very specific case of a dedicated application between a transmitter and a receiver that would be connected to each other by a private network in which specific optimizations would be implemented to optimize the targeted artificial intelligence task. Such an approach will not be able to "scale up" and generalize to the countless uses expected in artificial intelligence and more generally to all uses in which we seek to adapt the transmitted data according to the processing that will be carried out by the receiving application.

[0027] The invention improves the situation.

[0028] Statement of the invention

[0029] According to a first functional aspect, the invention relates to a method for managing the communication of vector data, called data to be communicated, from a first device to a second device, the method comprising:

[0030] - obtaining an index, called the obtained index, by applying an indexing function to the data to be communicated, the indexing function calculating an index from a vector data item, the indexing function being such that, the closer two vector data items are according to a distance measured between them, the closer the indexes calculated for these two vector data items by the function are to each other, said obtained index making it possible to find in a data structure another vector data item, called communicated data item, the data structure associating indexes with vector data, the obtained index being capable of being transmitted from the first device to the second device. In one embodiment, the method further comprises:

[0031] - transmit the index obtained from the first device to the second device.

[0032] Thanks to the invention, the number of symbols to be transmitted to communicate vector data is greatly reduced. Indeed, instead of transmitting the symbols or bits encoding the vector data as they are, the method obtains and then, in certain modes, transmits an index generally corresponding to an element of a data structure. It is not necessarily the same entity that obtains the index as the one that transmits it. This index makes it possible to find vector data in the data structure. If we assume that the vector data to be communicated is a vector of dimension 4096 of real numbers, and that these real numbers are represented by 32-bit floating points, its size is approximately 16 kilobytes. If the data structure comprises 65,536 elements, each of these elements can have a different index if they are sequences of 16 bits (indeed, 2 to the power of 16 is worth 65,536).Instead of transmitting 16 kilobytes, the process will therefore transmit 16 bits, or 2 bytes. The reduction in the size of the transmitted data is very spectacular here.

[0033] The data structure can be of any type as long as it allows vector data to be retrieved from an index. For example, it could be a key-value table in which the indexes act as keys and the values ​​are specific vector data. This is also called a dictionary. But the structure can also be a database. Or the structure can organize the indexes according to a prefix tree and the data stored in the structure will then be leaves of the tree in question.

[0034] The method includes obtaining a first index from the vector data to be communicated. The data structure includes a finite number of elements while the possible space of vector data is very large, even infinite. For example, if the data to be communicated are vectors of real numbers of dimension 4096, they belong to the vector space y? 4096 . In reality, the space of vector data is finite since it is computer data that are sets of floating numbers coded on 32 bits. However, it is reasonable to consider the possible space of vector data as infinite in first approximation. There is therefore very little chance that the data to be communicated, which belongs to an infinite space, is present in the data structure which is a finite set, of reasonable size.

[0035] The index obtained is capable of being transmitted from the first device to the second device. Instead of the data to be communicated, the method transmits (or has transmitted by another entity) an index which makes it possible to find another data item, which will therefore be the data actually communicated. In order for the semantic content associated with the data to be communicated to be preserved, the index obtained must make it possible to find in the data structure a vector data item whose associated semantic content is very close, or even as close as possible, to that of the data to be communicated. The index is therefore obtained by applying an indexing function to the data item to be communicated, a function which preserves the distance between vector data. The index is therefore chosen according to a distance measured between the distance to be communicated and the communicated distance. In general, the index will be chosen to minimize the distance measured between the data item to be communicated and the data actually communicated.

[0036] Two cases are then possible:

[0037] • Or the index obtained is exactly an index associated with a semantic data item in the data structure. In this case, if we assume that the index is transmitted without transmission error, it will allow us to directly find a semantic data item in the data structure, which is then the communicated data item.

[0038] • Or the index obtained by applying the indexing function to the data to be communicated is not an index associated with semantic data in the data structure. In this case, it can still be transmitted directly and it will be necessary, upon reception, to search in the data structure for a close index which will give access to semantic data. It is also possible to search, before transmission, for an index close to the index obtained, an index associated with vector data which will be transmitted.

[0039] In all cases, the index obtained makes it possible to find, directly or indirectly, vector data in the data structure, and it is this vector data which will be considered as being the communicated data.

[0040] It can be noted that the index obtained is not always directly associated with a semantic data item in the data structure but allows to find a semantic data item in the data structure. In addition, transmission errors added by the communication channel can cause an index obtained which was directly associated in the data structure with a vector data item to be transformed into an index which no longer corresponds directly to a vector data item. It must therefore in all cases be foreseen at the reception that it may be necessary to search in the data structure for an index close to the received index which corresponds directly to a vector data item in the data structure. The way in which the index is obtained must seek to ensure that the index obtained, to which transmission errors are added, allows to find in the data structure a vector data item which is close, according to the measured distance, to the data item to be communicated.

[0041] The data structure performs a quantization of the vector space in which the vector data takes values. Instead of an infinite number of possible values, a finite and restricted set (let's say it includes 65,536 elements) is defined which will give representatives for the data to be communicated. These representatives are chosen according to a measured distance between vector data. Two choices are possible concerning the size of the indexes with respect to the number of elements present in the data structure:

[0042] • Or the size of the indexes is such that the number of possible indexes corresponds exactly to the number of elements in the data structure. In the example above of 65,356 elements, a 16-bit index will give exactly this number. Each index obtained from a semantic data to be communicated will therefore give an index present in the data structure, associated with a communicated data, the index being chosen according to the distance between the data to be communicated and that communicated. This index will be transmitted, possibly with transmission errors, which will give another index which will also be present in the data structure. Assuming that these errors are few, the received index will make it possible to find in the data structure a received vector data which will also be at a suitable distance from the communicated data and the data to be communicated.

[0043] • Or the size of the indexes is larger than the size that gives a number of possible indexes equal to the number of elements in the data structure. In the example of 65,356 elements, we could choose a 20-bit index that takes more than a million possible values. In this case, there are many more possible indexes than values ​​in the data structure. The index obtained could be exactly the index of a data item present in the structure, or a close index. Transmission errors could cause the received index to drift from the transmitted index. But since there are many more possible index values ​​than elements present in the data structure, it is likely that the received index does not correspond to any element in the data structure but that the index closest to the received index is the one originally transmitted.

[0044] In other words, the length of the indexes and the size of the data structure can be such that the number of possible values ​​of the indexes is equal to the number of elements in the data structure or that the number of possible values ​​of the indexes is greater than the number of elements in the data structure. The transmission of the index will allow, upon reception, to find in the structure a vector data which will then be considered as being the communicated vector data. When the number of possible values ​​of the indexes is greater than the number of elements in the data structure and the received index is outside the data structure, it is the index closest to the received index which will allow to find a vector data which will be used.

[0045] In the first case, the advantage is to transmit an index of a minimal size compared to the number of elements present in the data structure. In the second case, the advantage is to be resistant to errors in transmitting the index.

[0046] The index is a sequence of symbols, belonging to a set of symbols. In some embodiments, these symbols are bits, that is to say they can take two possible values ​​generally represented by '0' and '1'. In other embodiments, these symbols correspond to quantities which will be those used in the physical transmission used by the method. For example, as explained above, the calculated index can be a sequence of symbols which correspond to phase change modulations among four possible phase changes, that is to say 4-PSK symbols. In this case, the indices would be sequences of values ​​chosen from the four possible 4-PSK values.A sequence of 8 4-PSK symbols will then take a value from among 65,536 possible values ​​(in fact, 4 to the power of 8 is worth 65,536) and a dictionary of 65,536 points can be indexed by indices of length 8 formed by 4-PSK symbols instead of indices of length 16 bits. From now on, we will most often choose examples in which the indices are sequences of bits but it is clear that these examples are only examples among others.

[0047] The transmission of the obtained index can be carried out by directly transmitting the sequence of symbols forming the index. In other embodiments, the sequence of symbols forming the obtained index is translated into a format suitable for transmission and then transmitted, which corresponds to the transmission of the index. In embodiments, it is the entity that obtains the index that will transmit the obtained index. In other embodiments, the index suitable for transmission is entrusted to another entity that will actually carry out the transmission.

[0048] We have seen that the reduction in the size of the communicated data is very strong, since instead of transmitting the symbols or bits forming the data as they are, we transmit an index that allows us to find a vector data item. Either the index obtained is associated with a vector data item in the data structure, and the associated data item is then found directly. Or the index obtained is not associated with a vector data item, but an index close to the index obtained, associated with a vector data item, can be obtained from the first index, so that a vector data item is then found indirectly. An advantage of the method is that this reduction is obtained without taking into account the subsequent processing that will be carried out on the communicated vector data. The transmission method is agnostic with respect to artificial intelligence applications or others using the vector data.This feature is particularly valuable for a telecommunications operator that wants to offer general data communication services to all kinds of possible customers.

[0049] The transmitted index is used to find a vector data item present in the data structure. The index is chosen using an indexing function that takes into account a distance measured between this vector data item and the initial vector data item to be communicated. The distance between vector data items can be measured by a distance in the mathematical sense of the term on the set to which the communicated data item belongs. For example, the vector data item is a vector of 4096 real numbers and the distance will be measured by a distance on the vector set y? 4096 . If the vector data is a vector of 1024 complexes, the distance will be measured by a distance on the vector set C 1024. Since what is transmitted instead of the data is an index corresponding to another vector data, index calculated by an indexing function which takes into account the distance between vector data in the set to which the data belongs, the distortion caused by the transmission of the index and by the recovery of a data present in the data structure instead of the initial data to be communicated itself, this distortion is minimized.

[0050] According to a variant of the invention, the data structure comprises a vector data item associated with all possible values ​​of an index of a given length.

[0051] In this way, when the indexing function calculates an index from the vector data to be communicated, the method is guaranteed to find in the data structure the index resulting from this calculation. This index can then be transmitted with the assurance that a vector data is indeed associated with it in the data structure.

[0052] According to a variant of the invention, the method comprises transmitting the calculated index from the first equipment to the second equipment.

[0053] In this variant, the index calculated by the indexing function from the vector data to be communicated is transmitted even if it is not associated with a vector data in the data structure. It will be at reception that the received index will make it possible to find an associated vector data in the data structure. The received index may be the calculated index that was transmitted if there are no transmission errors. There is therefore not necessarily a vector data item associated with the received index in the data structure but it will be possible to find the index closest to the received index for which a vector data item is associated. The received index may be different from the calculated index that was transmitted following one or more transmission errors. It will also be possible to obtain an index associated with a vector data item in the data structure by searching for the index that is closest to the received index.

[0054] According to a first embodiment of this functional aspect of the invention, the distance measured between the data to be communicated and the communicated data is a function of the proximity between the semantic contents associated with the two vector data to be communicated and communicated.

[0055] The choice of the distance used to measure the distance between vector data is made in such a way as to correspond to the subsequent interpretation of the semantic content of the data, that is to say that a minimum distance between vector data corresponds to a minimum distance between meanings interpreted by users and applications that manipulate the vector data. The measured distance is therefore a function of the proximity between the semantic contents associated with the vector data or more precisely is calculated by a function preserving the proximity between the semantic contents associated with the vector data. This function can be a distance in the mathematical sense of the term on the vector space in which the data take values.The fact that the function which measures the distance between vector data preserves the proximity between semantic contents associated with the data and that the index is chosen according to this distance measured between the data to be communicated and that actually communicated will ensure that the semantic content associated with the vector data to be communicated will not be too distorted by the distortion which consists of communicating another vector data, corresponding to the index obtained, in place of the data to be communicated.

[0056] The semantic content associated with vector data is that which is present at the origin of the production of data for later use. It can be the meaning of a text, an image, a video, a sound as this content will be interpreted by users or artificial intelligence applications. Such semantic content can be said to be encoded in the vector data, or present in the vector data, or integrated in the vector data, or represented by the vector data. We use here the term associated with the vector data.According to a second embodiment of this functional aspect of the invention, which may be implemented alternatively or cumulatively with the previous embodiment, the index corresponding to the data to be communicated is obtained by searching in the data structure for the vector data closest to the data to be transmitted according to the measured distance, and the transmitted index is the index associated with said closest vector data.

[0057] With this embodiment, a simple way to perform the index calculation is given. The vector data present in the data structure have been chosen in advance to perform an efficient quantization of the space in which the data to be transmitted are located. Indexes are associated with the vector data present in the data structure and have been calculated beforehand. The vector data present in the data structure closest to the vector data to be transmitted is determined, the proximity between the vector data being given by the measured distance between vector data. In general, this measured distance is such that a minimal distance between data corresponds to a minimal difference in subsequent semantic interpretations. In particular, the distance can be measured by a function preserving the proximity between the semantic contents associated with the vector data.The index associated with the closest data is found in the data structure and it is this index that will be transmitted subsequently.

[0058] In this way, it is not necessary to calculate an index for each vector data to be transmitted but simply to find in the data structure the vector data closest to the data to be transmitted.

[0059] According to the invention, the index is obtained by using an indexing function chosen in such a way that, for two distinct vector data, the closer these vector data are according to the distance measured between them, the closer the indexes calculated for these two vector data by the indexing function are to each other.

[0060] In this way, the method for communicating semantic data according to the invention resists possible transmission errors of the symbols that form the calculated index. Indeed, if one or more symbols, in small quantity, that form the index transmitted instead of the vector data, are changed during transmission, the index finally received will still be close to the index initially transmitted because the number of transmission errors is generally low. Indeed, if the transmission errors become too large, the transmission stops. The point in the dictionary that corresponds to the received index will therefore be close to the point that corresponds to the index initially transmitted by virtue of the choice of the indexing function.The distortion due to the transmission error of one or more symbols of the index, which is added to the distortion of the replacement of the vector data to be communicated by another data present in the data structure, will therefore also be minimized.

[0061] It can be noted that if a received index, due to transmission errors, does not correspond to an index present in the data structure, the receiver of the index will look for an index close to the received index which corresponds to an element of the structure. This search can make it possible to return to the index which was initially transmitted provided that the number of transmission errors is low enough. And if the index found in this way is still different from the initial index, it still remains close to the initial index and the choice of the indexing function then ensures proximity between the semantic data corresponding to the received index and the semantic data corresponding to the initial index.

[0062] According to another embodiment, which may be implemented alternatively or cumulatively with the previous embodiments, the distance between vector data is measured using cosine similarity.

[0063] Cosine similarity is a known quantity in the field of artificial intelligence to measure the similarity of data presenting a semantic content between them. The cosine similarity of two vectors is calculated by dividing the scalar product of the two vectors by the product of the norms of the two vectors. This quantity is between -1 and 1, and will be -1 for opposite vectors, 0 for orthogonal vectors and 1 for vectors which have the same direction, and which are therefore equal to a proportional factor.

[0064] It is possible to calculate a distance between vectors from the cosine similarity, for example by removing the cosine similarity from 1. Such a distance varies between 0 (for vectors having the same direction) and 2 (for opposite vectors). Other formulas for obtaining a distance from the cosine similarity are possible.

[0065] Such a distance between vectors has the advantage of not taking into account the magnitude of the vectors but only their direction. Such a distance is suitable for calculating semantic proximity between vector data integrating semantic content. It preserves the proximity between the semantic contents associated with the vector data whose distance is measured with a cosine similarity.

[0066] In other embodiments, the distance between two vector data is measured with the Euclidean distance.

[0067] The well-known Euclidean distance in vector spaces provides an alternative realization compared to distances using cosine similarity. Proximity according to Euclidean distance corresponds to a certain proximity between semantic contents associated with vector data.

[0068] According to another embodiment, which may be implemented alternatively or cumulatively with the previous modes, the proximity between two indexes is defined by the Hamming distance.

[0069] The Hamming distance is a defined distance between bit sequences that is calculated by counting the number of distinct bits at a given position between two bit sequences. For example, the Hamming distance between byte 00000000 and byte 00010001 is two, because, at the fourth and eighth positions, the bits are different. The notion of Hamming distance can be extended to apply to sequences of symbols as are the calculated indices. This distance provides a practical and efficient way to calculate the proximity between the calculated indices used in the method. The proximity between indices measured according to the Hamming distance corresponds well to the notion of seeking to minimize the distortion caused by transmission errors. Indeed, it is directly associated with the number of transmission errors suffered by a sequence of bits or symbols in a communication channel.

[0070] According to another embodiment, which may be implemented alternatively or cumulatively with the preceding embodiments, the indexing function is chosen in such a way that, for two distinct vector data, if the distance measured between these two vector data is less than a given threshold, then the probability that the indexes calculated by the indexing function for these two vector data are the same is greater than a given probability.

[0071] With this embodiment, the link between the distance measured between two vector data and the proximity between the indexes calculated for these two data is refined. In a bubble around a vector data, that is to say for the set of vector data which are at a distance lower than a threshold from the initial data, the probability that these data have the same calculated index is higher than a given probability. In this way, the closer the data are to each other, the closer their indexes are. The distances between vector data and proximities between indexes can be calculated using cosine similarities on the one hand, Hamming distances on the other hand, or any other suitable mode of calculating distances between vector data and indexes.

[0072] According to another embodiment, which may be implemented alternatively or cumulatively with the previous embodiments, the indexing function is a locality-sensitive hash function.

[0073] A locality-sensitive hash function (for which the acronym LSH can be used) is a function, which belongs to a family of functions, which calculates indexes for points of a given vector space and which verifies the following two properties, namely that, on the one hand, if the distance between two points is less than a threshold, the probability that the indexes calculated by this function for the two points are equal is greater than a first given probability and that, on the other hand, if the distance between two points is greater than another threshold, a multiple of the first, the probability that the indexes calculated for the two points are equal is less than a second given probability, much lower than the first.Functions of this type which index the points of a given vector space respond well to what is desirable for the invention, namely that close vector data will have close, or even merged, indexes, and that distant vector data will on the contrary have distant indexes, knowing that the vector data are points of a given vector space.

[0074] According to another embodiment, which may be implemented alternatively or cumulatively with the preceding embodiments, the indexing function is a locality-sensitive hash function for cosine similarity.

[0075] In this embodiment, the family of locality-sensitive hash functions is determined for a given distance between vector data and this distance uses cosine similarity. The advantage of this mode is to perform indexing adapted to this distance which is adapted to the proximity of the semantic contents associated with the vector data.

[0076] According to another embodiment, which may be implemented alternatively or cumulatively with the preceding embodiments, the method comprises a prior step of fragmenting the vector data to be communicated into several fragments and the subsequent steps of the method apply to said several fragments.

[0077] Until now, the method according to the invention processed a data item which is a point in a high-dimensional vector space (up to 4096 or even more) in a single block. But it may be advantageous to fragment the vector data item to be communicated and to apply the different steps of the method to data fragments. For example, a data item which is a vector of 4096 real numbers could be divided into 1024 fragments of dimension 4 or into 512 fragments of dimension 8.

[0078] A first advantage is to facilitate the exchange of data structures in certain embodiments of the method. Indeed, the data structure must be known in transmission and reception and, in certain cases, it will be transmitted before the implementation of the method. If we take our example of a data structure comprising 65,536 elements, and the elements are data in the vector space Æ 4096, then each element has a size of approximately 16 kilobytes, and the data structure has a size of approximately 1 gigabyte. Exchanging such a data structure between sender and receiver becomes prohibitive. If we fragment the semantic data into fragments of dimension 4, and keep the same size for the data structure, the 65,536 elements will weigh only 4 times 32 bits (i.e. 4 real numbers encoded in float32), that is to say that the structure will have a size of approximately 128 bits times 65,536, or 1 megabyte. The data structure that must be stored and possibly exchanged has therefore decreased in size by a factor of 1000.

[0079] Furthermore, fragmentation allows for an improvement in the precision of the communicated data, although it is true that it is at the expense of the number of symbols or bits transmitted forming the indexes. If we take the example of semantic data belonging to the vector space Æ 4096fragmented into 1024 fragments of dimension 4, and that the data structure used has 65,536 elements of dimension 4, these elements will be used to quantify fragments of dimension 4 and no longer a complete data of dimension 4096. The data communicated is then much more precise, to the detriment of the memory size required by the transmission of the indexes, since it will be necessary to transmit 1024 indexes of size 16 bits and no longer a single index of size 16 bits. The transmission gain of an index of size 16 bits instead of a vector data of dimension 4 (i.e. 4 reals encoded on 32 bits, therefore 128 bits) is therefore no more than a factor of 8.Fragmentation therefore allows a choice between very economical communication (a 16-bit index instead of data of approximately 16 kilobytes, i.e. a gain factor of 1024) or much more precise communication (65,536 elements to cover a space of dimension 4 instead of a space of dimension 4096) with a gain factor of only 8.

[0080] It is possible to mix these effects. While a data structure of size 65,536 elements can be seen as appropriate for data in the vector space Æ 4096 , fragmentation in a 4-dimensional space can allow for data structures of size 1024. The indexes for this structure size will then be 10 bits long. The gain factor compared to 4 real numbers encoded on 32 bits is then greater than 12. The precision given by 1024 elements can be quite sufficient to represent fragments of 4-dimensional vector data.

[0081] These digital examples are of course only some of the embodiments.

[0082] According to an embodiment which can be implemented cumulatively with the previous embodiment, the size of the fragments into which the vector data to be communicated is fragmented is chosen according to the expected size of the data packets which can be transmitted between the first device and the second device.

[0083] In this embodiment, fragmentation makes it possible to adapt the size of the transmitted indexes to the expected size of the packets that can be transmitted by the network. It is indeed advantageous not to divide a data item into several packets. The size of the packets that will be transmitted through the communication channel between the first and second devices is called the maximum transmission unit (MTU). Depending on the expected MTU on the transmission, a given fragmentation can be defined which will prevent the transmitted indexes from being divided into several packets.

[0084] According to another embodiment, which may be implemented alternatively or cumulatively with the previous embodiments, the data structure is defined beforehand according to the format of the vector data to be communicated.

[0085] The operation of the method implies that the data structure is available at the reception to find a vector data corresponding to the received index, which will be close to the data to be communicated by virtue of the distance measured between vector data and the chosen indexing function. It is possible to define in advance a data structure comprising vector data representative of the data to be communicated. This representative vector data will have the same format as the data to be communicated and will for example be of the same dimension.

[0086] One way to proceed is to define in advance a grid of points (the vector data present in the data structure) that appropriately covers the vector space in which the data to be communicated will take values. This grid can be denser at the locations of the vector space in which vector data to be communicated are expected in order to ensure a finer representation in the locations of interest. It can be noted that, in general, vector data used or produced by artificial intelligence applications have a reduced magnitude; the value of the real numbers composing a vector data to be communicated is most often between -100 and 100. It will therefore be advantageous for the vector data composing the data structure to be mostly around 0, with possibly larger values ​​to capture certain exceptional data.Such a data structure can therefore be defined in advance for a given vector data format. Applications handling vector data of a given format can then have in memory the data structure corresponding to this precise format. The method can then be used for any vector data communication task presenting a semantic content, by using the data structure associated with the format, and the receiving applications will be able to have information regarding the vector data format which will allow them to find the similar vector data associated with a received index, these similar data being present in the data structure.

[0087] This data structure defined for a specific vector data format can also be used to achieve efficient data storage. An index is calculated for a vector data to be stored, which corresponds in the data structure to a data close to the vector data to be stored. It is this index which is saved and which, upon subsequent reading, will give access to the data close to the vector data to be stored. Possible errors in saving the index will be minimized, as for transmission errors, by the nature of the chosen indexing function. And the distortion between the vector data to be stored and that corresponding to the index actually stored will also be minimized by the choice of the distance measured between vector data which preserves the proximity between semantic contents associated with the vector data.

[0088] According to another embodiment, which may be implemented alternatively or cumulatively with the previous embodiments, the data structure is obtained by automatic learning based on the vector data already communicated and / or intended to be communicated.

[0089] In this embodiment, the data structure is learned, by a machine learning method, in order to best adapt to the set of vector data already communicated or intended to be communicated. Several learning techniques can be used, for example supervised or not, or using or not using a reward function in the case of reinforcement learning. A machine learning technique will make it possible to have a greater density of representative data in the portion of the vector space in which a greater number of vector data to be communicated is expected.For the same size of the data structure, this embodiment will therefore have the advantage of greater proximity of the data communicated with the initial data to be communicated since the grid of communicated data is denser where it will be most needed, whereas a generic data structure, defined in advance, will be less precise.

[0090] According to another embodiment, which may be implemented alternatively or cumulatively with the previous embodiments, the automatic learning of the data structure uses the k-means method where k is the number of vector data present in the data structure.

[0091] When the embodiment of the method comprises automatic learning of the data structure, the objective of the automatic learning is to carry out a clustering of a set of data communicated or to be communicated, called the training set, around a reduced number of representative vector data which will be recorded in the data structure. Such a clustering can be carried out using the k-means method, where k is the size of the data structure. Thanks to this embodiment, the clustering is uniform, in the sense that the data belonging to the training set are well distributed in an egalitarian manner into groups whose center is a representative vector data which will be found in the data structure to represent all the vector data of the same clustering.

[0092] According to another embodiment, which may be implemented alternatively or cumulatively with the previous modes, the indexing function is defined beforehand according to the format of the vector data to be communicated.

[0093] In this embodiment, similarly to what was seen previously for the data structure, it is the indexing function that is defined beforehand according to the format of the vector data to be communicated. For a given vector space in which the data to be communicated will take values, an indexing function having the expected properties can be defined once and for all. This function can then be used by default for the vector data of the expected format. This embodiment has the advantage, when used in common with the similar mode relating to the data structure, of allowing efficient communications, without prior processing, of the vector data of a given format. These communications can also correspond to backups as we have already seen.

[0094] According to another embodiment, which may be implemented alternatively or cumulatively with the previous embodiments, the indexing function is obtained by automatic learning based on the vector data already communicated and / or intended to be communicated.

[0095] This embodiment has the advantage of allowing the use of an indexing function particularly suited to the vector data to be communicated. For example, as for learning the data structure, the learned indexing function may be more discriminating in areas of the vector space where the density of vector data to be communicated is higher.

[0096] Some functions are particularly suited to machine learning. This is the case of locality-sensitive hash functions (LSH functions). These functions are defined according to hyperplanes that divide the vector space in which the data to be communicated take values ​​into two parts. One part, located on one side of the hyperplane, will correspond to the value 0 for a bit of given rank in the calculated index while the other part, located on the other side, will correspond to the value 1 for the bit of the same rank. For a given index length L, an equal number of hyperplanes is fixed that will divide the vector space into 2 to the power of L, if we consider that the index is a sequence of bits. Similar calculations can be done when the index is a sequence of other symbols, by reducing these other symbols to bits.The placement of these hyperplanes, which define an LSH indexing function, can be obtained by learning according to the density of the data to be transmitted in the vector space, in order to produce the most suitable cuts.

[0097] According to another embodiment, which may be implemented alternatively or cumulatively with the previous embodiments, the method comprises the transmission of parameter values ​​useful for the communication of vector data according to the method.

[0098] The method according to the invention operates by assuming that the entity receiving the transmitted index knows at least the data structure to find a vector data item corresponding to the received index. Other parameter values ​​useful for the operation of the method may need to be known by the receiving entity, for example the size of the fragments if the vector data item has been divided into several fragments before applying the method.

[0099] For example, when the data structure and the indexing function are defined beforehand, as has been seen in several embodiments, the transmitter simply needs to place in its transmission of the index obtained references to the data structure and to the function that it will use. These references can be placed for example in a preamble packet to the transmission or in a data packet header. The receiver will receive these parameter values ​​which refer to known indexing functions or data structures. It will then know which data structure to use to find the vector data close to the data to be transmitted thanks to the received index. Such data structures and parameter sets could be defined beforehand and standardized for vector data formats used in a generalized manner, such as those presented in the MTEB comparison.The communication of vector data according to the method would then comprise in this embodiment the transmission of reference information allowing the equipment receiving the index associated with the communicated vector data to know which parameters are used for the communication.

[0100] According to another embodiment, which may be implemented alternatively or cumulatively with the preceding embodiments, the method comprises a preliminary phase of negotiation of parameter values ​​useful for the communication of vector data according to the method, said parameters being able to be chosen from the following non-exhaustive list:

[0101] 1. Dimension of the vector data;

[0102] 2. Dimension of a fragment of the vector data;

[0103] 3. Data type used as the base type by a vector data;

[0104] 4. Vector data format;

[0105] 5. Identifier of the model producing the vector data;

[0106] 6. Dictionary size;

[0107] 7. Dictionary contents; 8. Size of dictionary indexes;

[0108] 9. Symbols used by the indexes;

[0109] 10. Indexing function used.

[0110] With this embodiment, more advanced possibilities of communication between the transmitter and the receiver are implemented. We saw in the previous embodiment that values ​​useful for the execution of the parameter could be transmitted, for example in a packet header, and make it possible to find elements already defined, such as the data structure used in reception to find the vector data close to the data to be communicated. But this previous mode is not sufficient when, for example, the data structure and / or the indexing function are learned by machine learning in a specialized way for data to be communicated. The receiver cannot in this case know a priori the elements necessary for the operation of the method.

[0111] In this embodiment, a negotiation phase prior to communication allows the transmitter and the receiver to agree on all the parameters useful for the communication of vector data according to the method. This mode can be combined with the previous one: for example, a first transmission can give the references to a codec which will be used, then a negotiation allows the transmitter and the receiver to agree on a fragmentation, which can depend on the MTU expected between the first device and the second device. This embodiment proposes a non-exhaustive list of the elements which can be negotiated during this preliminary negotiation step.

[0112] According to a second functional aspect, the invention relates to a method for managing the reception of vector data, characterized in that it comprises the reception of an index, called the received index, followed by the reading of vector data in a data structure associating indexes with vector data as a function of the received index.

[0113] This second functional aspect of the invention corresponds to the reception of the transmitted index in place of the vector data to be communicated. We speak here of received index, because it is possible that transmission errors have modified the index during transmission and that the received index is not equal to the transmitted index. Once the index is received, it is possible to find in the data structure a vector data from the received index. This read vector data will be considered as the vector data to be communicated initially.

[0114] According to a first embodiment of this second functional aspect, the method comprises, if the received index does not correspond to any index present in the data structure, a search in the data structure for the index closest to the received index and the vector data read in the data structure is that which corresponds to said closest index.

[0115] According to another embodiment of this second functional aspect, which may be implemented alternatively or cumulatively with the previous mode, the proximity between the received index and the indexes present in the data structure is defined by the Hamming distance. As already indicated previously, transmission errors may cause the received index to be different from the transmitted index and in particular, the received index not to be present in the data structure. In this case, an advantageous embodiment is to search for the closest index that is present in the data structure in order to finally obtain vector data.

[0116] To summarize, the method according to the invention can accumulate the following distortions in the transmission of vector data:

[0117] • In general, the vector data to be communicated is not present in the data structure. The calculated index will correspond directly or indirectly to a close vector data present in the data structure, called communicated vector data. o In some cases, there is as much vector data in the data structure as there are possible index values. The calculated index will in this case be directly associated with a vector data. o In other cases, there is less vector data in the data structure than possible index values. Then, the calculated index is not necessarily directly associated with a vector data present in the data structure. But it is possible to find in the data structure the index closest to the calculated index. A vector data associated with this closest index can then be found in the data structure.o In all cases, the calculated index therefore corresponds directly or indirectly to a vector data present in the data structure. The indexing function and the proximity between indexes are defined so that this transmitted vector data is close to the data to be transmitted according to a measured distance which preserves the semantic contents associated with the vector data, but this is still a first distortion.

[0118] • The calculated index is transmitted whether or not it is associated with a semantic data item in the data structure. It can be modified by the transmission channel and the received index can be different from the transmitted index. o In some cases, there are as many vector data items in the data structure as there are possible index values. The received index will in this case be directly associated with a vector data item, which may be different from the communicated vector data item. This is a second distortion. o In other cases, there are fewer vector data items in the data structure than possible index values. Then, the received index is not necessarily directly associated with a vector data item present in the data structure. But it is possible to find in the data structure the index closest to the received index.This closest index can be the transmitted index: in this case, we fall back on the index that we wanted to transmit and the distortion due to transmission errors is canceled. Or the closest index is different from the transmitted index: in this case, another vector data than the communicated data is associated, and here we have a second distortion.

[0119] The method according to the invention must therefore make it possible to carefully choose the indexing function, the proximity between indexes, the fact that the distance measured between vector data preserves the semantic contents associated with the vector data, the number of representative data present in the data structure, the concentration of these representative data in the areas richest in data to be communicated from the vector space in which the data take values ​​and any other useful parameter in order to minimize these distortions.As indicated, the choice of cosine similarity, Hamming distance and locality-sensitive hash functions for cosine similarity are examples of operating parameters of the method according to the invention which give first good results in our experiments, but the method according to the invention is not limited to these exemplary embodiments and covers the use of many other possible parameters.

[0120] According to another embodiment of the second functional aspect of the invention, which may be implemented alternatively or cumulatively with the preceding modes, the method comprises, if the index received corresponds to a fragment of vector data, the reconstitution of complete vector data from the fragments of vector data corresponding to the indexes received.

[0121] Thanks to this embodiment, the method according to the invention makes it possible to manage the fragmentation of the communicated vector data. It is simply necessary to ensure that the complete vector data is reconstituted from the fragments of vector data corresponding to the different indexes received. This reconstitution makes it possible to complete the operation of the method according to the invention.

[0122] According to a first material aspect, the invention relates to a communication manager capable of managing the communication of vector data, called data to be communicated, from a first device to a second device, the communication manager comprising:

[0123] • An index obtaining module, the obtaining module being able to obtain an index, called the obtained index, by applying an indexing function to the data to be communicated, the indexing function calculating an index from a vector data item, the indexing function being such that, the closer two vector data items are according to a distance measured between them, the closer the indexes calculated for these two vector data items by the indexing function are to each other, said obtained index making it possible to find in a data structure another vector data item, called communicated data item, the data structure associating indexes with vector data, the index obtaining module being able to provide an index to an index transmitter able to transmit an index from the first device to the second device.

[0124] In one embodiment, the communication manager further comprises a transmitter of the index obtained from the first device to the second device. The communication manager may also be referred to as a management entity capable of managing the communication of data.

[0125] According to a first embodiment of this hardware aspect, the invention relates to telecommunications equipment comprising a communication manager capable of managing the communication of vector data according to the invention.

[0126] Such telecommunications equipment may be, for example, a router, or a home gateway or one in a professional environment, or a network card present in computer equipment such as a computer, or a mobile terminal, or a communicating object such as a camera or other object producing vector data for artificial intelligence applications or not. In general, a communication manager according to the invention may be embedded in any type of object or equipment producing vector data intended for a remote receiver or having to carry out the communication of such data intended for other equipment in a computer or telecommunications network.

[0127] According to a second hardware aspect, the invention relates to a vector data reception manager comprising:

[0128] • A module for receiving an index, called the received index;

[0129] • A reader of vector data into a data structure associating indexes with vector data based on the received index.

[0130] The reception manager can also be referred to as a management entity capable of managing the reception of data.

[0131] According to a first embodiment of this hardware aspect, the invention relates to telecommunications equipment comprising a reception manager capable of managing the reception of vector data according to the invention.

[0132] Such telecommunications equipment may be, for example, a router, or a gateway in a home or professional environment, or a network card present in computer equipment such as a computer, or a mobile terminal, or equipment managing the reception of data in a cloud computing system. In general, a reception manager according to the invention may be embedded in any type of object or equipment receiving vector data for saving or processing by artificial intelligence or other applications or having to carry out the reception of such data in a computer or telecommunications network.

[0133] In some embodiments, the communication and reception managers for vector data are integrated into a single device. In some of these embodiments, the devices in question may be routers performing both communication to other devices and reception from other devices of vector data. In other embodiments, the communication and reception managers correspond to separate components of the same device. The communication manager may correspond to a component producing vector data and the reception manager to a component for storing and reading the vector data produced. The transmission channel between the devices in this case is a software bus which connects the two components of the same device together.

[0134] According to another material aspect, the invention relates to a computer program capable of being implemented by a communication manager, the program comprising code instructions which, when executed by a processor, carry out the steps of the method for managing the communication of vector data defined above.

[0135] According to another material aspect, the invention relates to a computer program capable of being implemented by a reception manager, the program comprising code instructions which, when executed by a processor, carry out the steps of the method for managing the reception of vector data defined above.

[0136] Finally, according to another material aspect, the invention relates to data media on which are recorded computer programs comprising sequences of instructions for implementing the methods for managing communication and reception of vector data defined above.

[0137] The data carriers may be any entity or device capable of storing the programs. For example, the carriers may comprise a storage means, such as a ROM, for example a CD ROM or a microelectronic circuit ROM, or a magnetic recording means such as a hard disk. On the other hand, the carriers may be transmissible media such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means. The programs according to the invention may in particular be downloaded from a network such as the Internet. Alternatively, the information carrier may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the method in question.The program according to the invention can use any type of computer technology in terms of compiled, interpreted programming languages, or a combination of both as well as in terms of operating systems.

[0138] Brief description of the figures

[0139] The invention will be better understood on reading the following description, given by way of example, and made with reference to the appended drawings in which:

[0140] [Fig 1] represents a communication manager and a vector data reception manager, included in two devices, illustrating an exemplary embodiment of the invention. [Fig 2] illustrates an example of steps implemented within the framework of an embodiment of the invention.

[0141] [Fig 3] illustrates examples of possible distortions undergone by vector data during its communication and reception.

[0142] [Fig 4] illustrates experimental results obtained during the development of the methods according to the invention.

[0143] Detailed description

[0144] Figure 1 represents a communication manager 100 managing the communication of a vector data SD from a device EQ1 to a device EQ2, through a communication channel CNL. A reception manager 200 manages the reception of a vector data SD'.

[0145] In the exemplary embodiment presented in Figure 1, the management entity 100 comprises the following modules:

[0146] • A module 101 for obtaining an index I', the obtaining module 101 being able to obtain an index I' by applying an indexing function F to the data to be communicated SD, the indexing function F calculating an index from a vector data item, the indexing function F being such that, the closer two vector data items are according to a distance D measured between them, the closer the indexes calculated for these two vector data items by the function F are to each other, said index I' making it possible to find in a data structure DS another vector data item SD', called communicated data item, the data structure DS associating indexes with vector data;

[0147] • A transmitter 102 for transmitting the index I' obtained through a CNL communication channel between the first equipment EQ1 and the second equipment EQ2.

[0148] In the exemplary embodiment presented in figure 1, the index I' is obtained from the semantic data SD by using at one point an indexing function F, the indexing function F being such that, the closer two vector data are according to the distance D measured between them, the closer the indexes calculated for these two vector data by the function F are to each other.

[0149] In the exemplary embodiment presented in Figure 1, the application of the function F to the vector data SD directly produces an index I' such that there is a vector data SD' associated with the index I' in the data structure DS. In other exemplary embodiments, the indexing function F calculates from the data SD an index I which is not present in the data structure DS and an index I' close to the index I is obtained, a vector data being associated with the in the structure DS. In still other examples, the indexing function calculates from the vector data SD an index I and this index is transmitted even if it is not present in the data structure DS. It is upon reception that the closest index present in the data structure will be searched in order to find a communicated vector data corresponding to the transmitted index.In all cases, the index I' obtained and then transmitted makes it possible to directly or indirectly find vector data SD' in the data structure DS, which will be considered as being the vector data communicated instead of the data SD to be communicated.

[0150] In the exemplary embodiment presented in Figure 1, the management entity 200 comprises:

[0151] • A module 201 for receiving an index I', called the received index;

[0152] • A reader 202 of vector data SD' in a data structure DS associating indexes with vector data according to the received index I'.

[0153] Instead of directly transmitting the symbols forming the vector data SD, the management entity 100 obtains an index I' using an indexing function F and transmits it, which greatly saves the size of data to be transmitted.

[0154] The transmission of the symbols forming the index I' is done in the example of figure 1 through a CNL communication channel. Transmission errors can occur during transmission. However, in the example presented in figure 1, the received index I' is equal to the calculated index I'. Figure 3 shows examples in which transmission errors appear. The management entity 200 can read in a data structure DS a vector data item SD' corresponding to the received index I'. This vector data item SD' will then be used in subsequent artificial intelligence or other applications in place of the data item SD to be transmitted. The vector data item SD' was therefore communicated from the equipment EQ1 to the equipment EQ2 in place of the vector data item SD which was the data item to be communicated.

[0155] In the example presented in Figure 1, the only distortion caused by the use of the method according to the invention corresponds to the use of the calculated index I', associated with the vector data SD', present in the data structure DS. We present in Figure 3 all the distortions that can be introduced by the method according to the invention.

[0156] The communication manager 100 included in the equipment EQ1 has the hardware architecture of a conventional computer. It includes in particular a processor, a RAM type random access memory and a read-only memory such as a Flash, ROM type memory (not shown in the figure) as well as input-output devices such as, in certain cases, keyboards and / or screens (not shown in the figure), and network ports allowing communication with other entities and servers via a communication network such as the Internet, not shown in the figure.

[0157] The reception manager 200 included in the EQ2 equipment also has the hardware architecture of a conventional computer. It includes in particular a processor, a RAM type random access memory and a read-only memory such as a Flash, ROM type memory (not shown in the figure) as well as input-output devices such as, in certain cases, keyboards and / or screens (not shown in the figure), and network ports allowing communication with other entities and servers via a communication network such as the Internet, not shown in the figure.

[0158] The managers 100 and 200 can be included for example in telecommunications equipment. Such equipment EQ1, EQ2 can be routers, home gateways or deployed in professional environments. The management entities 100 and 200 can be included in network cards present in computer equipment EQ1, EQ2 such as computers or equipment EQ1, EQ2 managing the reception or transmission of data in cloud computing systems. The managers 100 and 200 can also be included in mobile terminals EQ1, EQ2 where they will be used respectively for the communication or reception of vector data through 3G, 4G, 5G, 6G or other wireless telecommunications networks to which the mobile terminals EQ1, EQ2 are connected.In general, a manager 100 or 200 according to the invention can be embedded in any type of object or equipment EQ1, EQ2 producing vector data SD for saving or processing by artificial intelligence or other applications or having to carry out the reception of such data in a computer or telecommunications network.

[0159] In another technical field, the managers 100 and 200 can be included in image and sound sensors and in all types of EQ1, EQ2 equipment, for example means of transport or in robotic EQ1, EQ2 equipment. Image or sound sensors can encode vector data from the captured images or sounds. Such sensors (cameras, microphones) can include a manager 100 for communicating vector data. Managers 200 can be deployed in processing servers which will receive the transmitted vector data and act on them. These managers 200 can for example be found in servers embedded in means of transport where they will be used for the interpretation of data transmitted by the sensors via the managers 100 for communicating vector data.This organization of managers 100 and 200 can be useful in the context of autonomous means of transport, or robots, in which sensors process captured images and sounds, produce vector data, and this vector data is sent for interpretation and consequent action to control servers. In this context, managers 100 and 200 can be included in the same equipment.

[0160] In another domain, the managers 100 and 200 can belong to the same equipment and manage the storage of vector data. In this context, the manager 100 obtains an index I' from a vector data item SD to be stored and it is this index I' which is stored. The manager 200 retrieves the stored index I' and reads the corresponding vector data SD' in the data structure DS. The vector data SD' is therefore ultimately the data which has been stored in place of the vector data SD to be stored.

[0161] The CNL communication channel is a link between the managers 100 and 200. The CNL channel may be a segment of a telecommunications network or, on the contrary, correspond to the chaining of several communication networks making it possible to connect the manager 100 to the manager 200. The CNL channel may also be a link inside a piece of equipment (autonomous means of transport, robot, computer equipment in general) between sensors (cameras, microphones) and processing servers or between a component producing semantic data and storing them and a component retrieving them. In these cases, the CNL channel may be a data bus inside a piece of computer equipment. The communications technologies deployed in the CNL channel may be wired, optical, satellite or wireless technologies or a combination of these different technologies.

[0162] The CNL channel can provide a direct connection between the manager 100 and the manager 200. For example, the manager 100 can be included in a network card of a personal computer; the manager 200 can be included in a home gateway deploying a local area network with WiFi technology; and the CNL channel can be in this case the WiFi connection between the computer including the manager 100 and the gateway including the manager 200. Or, the manager 100 can be included in a mobile phone using wireless telephony of 3G, 4G, 5G, 6G or other technology and the manager 200 can be included in a base station deploying the corresponding technology and the CNL channel will be the wireless connection between the mobile phone including the manager 100 and the base station including the manager 200.Or the manager 100 is associated with a camera, a microphone or a radar or lidar system and addresses vector data to the manager 200 included in a processing server embedded in an autonomous vehicle or a robot via an internal coaxial cable or optical fiber link, which forms the CNL channel. Or the CNL channel is a software or data bus connecting components of computer equipment.

[0163] In other examples, the CNL channel will provide an indirect link between the manager 100 and the manager 200. For example, the managers 100 and 200 may be included in routers EQ1, EQ2 present in packet-switched networks, which may be, for example, IP routers (acronym for Internet Protocol). The CNL channel may in this example be a path according to the IP protocol between the two routers EQ1, EQ2 which include the managers 100 and 200. In this exemplary embodiment, the CNL channel may therefore use several IP network segments and several IP routers and will then be an indirect link between the two managers 100 and 200.

[0164] The CNL channel can be seen, in embodiments, as transmitting sequences of bits, that is to say sequences of binary symbols 0 or 1 between the two devices EQ1, EQ2 including the two managers 100 and 200. But at the level of physical transmission, the CNL channel will transmit physical symbols which can take more values ​​than just two values. For example, in the field of wireless communications, the symbols transmitted by the CNL channel can be phase modulations carried by a radio wave, and these phase modulations can take up to sixteen or more distinct values. Similarly, in the field of links by optical fiber or copper media, the symbols transmitted by the CNL channel will be variations of electromagnetic waves carried by the physical media.The calculated index l' which is transmitted by the manager 100 through the CNL channel will therefore be in certain embodiments a sequence of bits 0 or 1 but will be, in other embodiments, a sequence of symbols corresponding to the symbols which can be transmitted in the CNL channel.

[0165] The CNL channel introduces errors into the transmission. The received index I' is therefore, in general, different from the calculated index I' that was transmitted. In the example shown in Figure 1, these two indexes are identical.

[0166] The manager 200 has a data structure DS which makes it possible to match the received index I' with a vector data SD'. This vector data SD' is then made available by the manager 200 and used by artificial intelligence or other applications in place of the initial vector data SD. Instead of communicating the complete vector data SD by transmitting all the symbols or bits constituting it, the manager 100 only had to transmit the index I', which represents a significant saving in the size of the transmitted data.

[0167] The data structure DS can be of any type. It can be, for example, a key-value table, or a dictionary, associating indexes I, I' with respective vector data SD, SD'. The data structure DS can also be a database where the indexes I, I' are associated with the respective vector data SD, SD'. The data structure DS can be saved in RAM or ROM by the management entity 200 or else queried on demand upon receipt of an index I' to retrieve the associated vector data SD'.

[0168] SD, SD' vector data are data obtained by transforming data such as texts or images into a model. The semantic content of the textual or image data is then associated with the obtained vector data. The fact that SD, SD' vector data preserves the semantic content of the initial data associated with them makes them particularly suitable for generative or other artificial intelligence applications. SD, SD' vector data generally follows a given format that will allow them to encode, integrate or represent more or less semantic information. Possible formats are listed on the MTEB website (acronym for Massive Text Embedding Benchmark) for SD, SD' vector data that represent texts and their performance in different artificial intelligence tasks is presented. A high-performance format is, for example, the Mistral-7B format, from the company Mistral Al.The vector data SD, SD' in this format are vectors of 4096 real numbers, i.e. elements of the vector space y?. 4096 . Equivalently, we can say that the vector data SD, SD' are points belonging to the Cartesian space of the same dimension y? 4096 . Other formats can be used, for example 1024 dimension.

[0169] The size of an SD vector data item according to the Mistral-7B format is therefore 4096 times the 32 bits used to encode a real number in the float32 format. If the communication of an SD vector data item is replaced by the transmission of an index of length 32 bits, the gain in size of the transmitted data item is then a factor of 4096. Other smaller or larger gains can be obtained depending on the indexing choices made during the implementation of a possible embodiment of the invention from a whole set of possible embodiments.

[0170] The transmission of an index I' instead of the transmission of the symbols or bits forming the vector data SD, then the transmission errors caused by the CNL channel, and finally the reading of a data SD' corresponding to the received index I' involve distortions with respect to the vector data to be communicated SD. These distortions are presented in more detail in Figure 3. In the example of Figure 1, an indexing function F is chosen so as to minimize these distortions according to a distance D measured between vector data. In Figure 1, this is represented by a minimum distance D between the vector data SD and the vector data SD'.

[0171] The distance D is measured by a function measuring the proximity between two vector data SD and SD'. This function can be a distance, in the mathematical sense of the term, in the vector space to which the vector data SD, SD' belong. In embodiments, the distance D can be for example the Euclidean distance between points of the Cartesian space associated with the vector space of the vector data SD, SD'. In other embodiments, the distance D can be obtained by calculating the cosine similarity between vectors of the vector space of the data SD, SD'. Other distances can be defined which will be specific to the subsequent tasks which will use the vector data SD, SD' so that the proximity between vector data SD, SD' measured by the distance D is in direct relation with the proximity of the semantic contents associated with the data themselves in the subsequent task which uses them.

[0172] In embodiments, the distance D can be learned from a training set representative of the vector data SD, SD' that will have to be transmitted. Indeed, if the vector data SD to be transmitted are rather concentrated in a precise zone of the vector space to which they belong, it may be relevant for the function measuring the distance D to be more discriminating in this zone, to take into account differences between data concentrated in this zone, and less discriminating outside this zone since it is expected to find less vector data SD there.

[0173] We recall that the index l' is chosen as a function of a distance D measured between the vector data to be communicated SD and that communicated SD' and that, in embodiments, this measured distance D is calculated by a function preserving the semantic proximity between the semantic contents associated with the two vector data SD, SD'.

[0174] In the example shown in Figure 1, the indexing function F is chosen so as to ensure minimal distortion between the vector data to be transmitted SD and the transmitted data SD'. Furthermore, according to the invention, the indexing function F is chosen in such a way that for two distinct vector data SD, SD', the closer these data SD, SD' are to each other according to the distance D measured between them, the closer the calculated indices I, I' for these two vector data SD, SD' are to each other. The proximity between calculated indices can be determined by the Hamming distance or by another appropriate calculation method. An advantage of this choice is to ensure a certain robustness of the method in the presence of transmission errors in the CNL channel which affect the transmitted index.

[0175] The F function calculates an index of size limited to a maximum of a few bits (or symbols in the case where the CNL channel transmits different symbols on its physical layer) from a vector data item made up of several thousand real numbers. For example, we will use indexes of length 10 bits, which gives 1024 possible index values, or 16 bits, which gives 65,536 possible index values. These indexes are associated in a DS data structure with SD, SD' vector data. Given the size in number of bits of the vector data (16 kilobytes for a SD vector data item of size 4096 real numbers encoded in float 32), if the DS data structure includes 65,536 data items associated with all the possible values ​​taken by an index of length 16 bits, it will have a size of

[0176] I gigabyte, which could be seen as a maximum size.

[0177] It is therefore clear that the number of possible values ​​for indexes is very small compared to the number of possible values ​​for vector data.

[0178] Furthermore, it is not mandatory that there exists in the data structure DS a vector data associated with all possible index values. The calculated index I' from the vector data SD can therefore be obtained in two ways:

[0179] • Or a first index I is initially calculated from the vector data SD using the indexing function F, but this index is not present in the data structure DS. Two cases can then arise depending on the embodiments: o The method can search in the data structure DS for an index I' close to the initially calculated index I and it is this index I' which will be the index obtained by the module 101 and finally transmitted by the module 102 in the CNL channel. The proximity between indexes could for example be defined by the Hamming distance and must in any case ensure the proximity between the data to be transmitted SD and the transmitted data SD' associated with the calculated index I' in the data structure DS. o Or the index I obtained from the vector data SD is not present in the data structure DS but is nevertheless transmitted by the module 102 in the CNL channel.It is the management entity 200 of the reception which will have to search from the received index (I, in the case where the CNL channel does not introduce transmission errors) a close index present in the DS data structure, which will make it possible to read in the DS data structure a vector data associated with this close index.

[0180] • Or the index obtained from the vector data SD is present in the data structure DS, for example because the latter covers the set of all the values ​​that an index of given length can take. In this case, the method transmits the obtained index I' but this corresponds, in all probability, in the data structure DS, to a vector data SD' different from SD. The indexing function F or another method of obtaining the index I' must then ensure proximity between the vector data SD to be communicated and the vector data SD' actually communicated via the obtained index I'.

[0181] In embodiments, the indexing function F is chosen such that, for two distinct vector data SD, SD', if the given distance D between these two data SD, SD' is less than a given threshold, then the probability that the calculated indexes I, I' by the indexing function F for these two vector data SD, SD' are the same is greater than a given probability. The choice of an indexing function F having this property makes it possible to increase the chances that two vector data close according to the distance D measured between them will have the same calculated index.

[0182] An indexing function F with these properties is given by the family of locality-sensitive hashing functions (LSH functions). These functions produce indexes that are close to each other, or even equal, for starting elements that are close to each other for a distance D. Other examples of functions with this property can be used in different embodiments.

[0183] The indexing function F chosen may be an LSH function for cosine similarity. In other embodiments, it may be an LSH function for Euclidean distance.

[0184] In embodiments, the calculation of the index I' is carried out by searching in the data structure DS for the vector data SD' closest to the data to be transmitted SD according to the measured distance D. In this way, it is ensured that the data transmitted SD' via the calculated index I' is indeed the closest vector data SD' among the possible data present in the data structure DS.

[0185] Here, the indexes present in the data structure DS have been obtained beforehand for a set of vector data present in the data structure DS which achieve an appropriate coverage of the values ​​which can be taken by the vector data SD to be communicated in the vector space to which they belong. In this way, it is not necessary to calculate for each vector data SD to be transmitted an index I' but simply to find in the data structure DS the vector data SD' closest to the data to be communicated SD according to the distance D, then to transmit this index l' found in the data structure DS.

[0186] The DS data structure therefore performs a quantification of the vector space in which the vector data SD, SD' to be communicated can take values. This quantification makes it possible to transmit, instead of the bits or symbols constituting a complete SD vector data item, an index I' calculated by the function F, an index which is associated in the DS data structure with a SD' vector data item close to the SD vector data item according to the distance D.

[0187] In general, the methods according to the invention operate when the data structure DS is known by both the communication managers 100 and the reception managers 200. This is not an obligation: the communication manager 100 can simply calculate an index I using the indexing function F for a vector data item SD and transmit it, without consideration of the presence of the index I in a data structure DS, and it is at reception that the manager 200 will make the connection between the received index (I, if the CNL channel does not introduce transmission errors, or I' if there have been transmission errors) and those present in the data structure DS.

[0188] In exemplary embodiments, the method for managing the communication of vector data SD by the manager 100 comprises a fragmentation step.

[0189] We have seen that the DS data structure performs a quantization of the vector space in which the vector data SD, SD' take values. But the size of the DS data structure is limited. We have seen, for example, that a DS data structure associating 65,536 vector data SD, SD' with a size of 4096 reals encoded on 32 bits would have an overall size close to a gigabyte. Such a size would probably be maximum. It may therefore be interesting instead to fragment the vector data SD to be communicated, for example into fragments of dimensions 4, 16, 64, etc. The DS data structure used then would associate fragments of vector data of the given dimension with indexes. The same principles regarding the indexing calculation relative to the given distance D then apply, but in the vector space of fragments, of reduced dimension compared to the complete vector space of the vector data.The index I' obtained is then an index making it possible to find in the data structure DS a fragment of given dimension. In associated exemplary embodiments, the method of receiving vector data by the reception manager 200 then comprises a defragmentation step. The reception manager 200 receives in these embodiments several indexes corresponding to the different fragments, indexes transmitted by the communication manager 100. After having found in the data structure DS the fragments corresponding to the received indexes, the reception manager 200 reconstructs a complete vector data by combining the different fragments found.

[0190] In embodiments, the DS data structure is predefined based on the format of the SD vector data to be transmitted.

[0191] For a format frequently used in artificial intelligence applications, it may be interesting to define in advance one or more DS data structures that perform an adapted quantization of the vector space corresponding to the vector data to be transmitted. For a given format, several different DS data structures may exist that will be more or less adapted to the applications using the communicated vector data. Vector data corresponding to visual content may, for example, be more concentrated in a specific area of ​​the vector space where they take values, different from the areas where vector data corresponding to textual content are found, and this for the same vector data format. The interesting quantizations, and the DS data structures corresponding to these two tasks will therefore be distinct.For a given vector data format to be communicated, the DS data structure can be defined using a machine learning algorithm. Indeed, the best way to concentrate the quantification performed by the DS data structure where it is useful is to perform machine learning from a corpus of vector data. This corpus can be representative of a specific task using the vector data to be transmitted, or it can be generic for all types of vector data using a given format.

[0192] In the case of machine learning of the DS data structure, a suitable learning algorithm is the k-means method where k is the number of vector data present in the data structure. Here, machine learning will perform clustering of a set of SD, SD' data communicated or to be communicated, which form a training set, around a reduced number of representative vector data that will be stored in the DS data structure. Such clustering can be performed using the k-means method, where k is the size of the data structure.With this embodiment, the clustering is uniform, in the sense that the data belonging to the training set are well distributed egalitarianly into groups whose center is a representative vector data which will be found in the DS data structure to represent all the vector data of the same clustering.

[0193] Relatedly, an indexing function can be learned that will index each vector data present in the DS data structure so that data close according to the given distance D present in the DS data structure will be associated with indexes close to each other, for example according to the Hamming distance between indexes.

[0194] In embodiments, the indexing function F is predefined for a given vector data format or learned according to a training corpus. The training corpus may be generic, for all tasks expected of vector data in a given format, or specific to the vector data to be communicated for a given task.

[0195] Some functions are particularly suited to machine learning. This is the case of locality-sensitive hash functions (LSH functions). These functions are defined according to hyperplanes that divide the vector space in which the vector data take values ​​into two parts. One part, located on one side of the hyperplane, will correspond to the value 0 for a bit of given rank in the calculated index while the other part, located on the other side, will correspond to the value 1 for the bit of the same rank. For a given index length L, an equal number of hyperplanes is fixed that will divide the vector space into 2 to the power of L, if we consider that the index is a sequence of bits. Similar calculations can be done when the index is a sequence of other symbols, by reducing these other symbols to bits.

[0196] The placement of these hyperplanes, which define an indexing function LSH, can be obtained by learning according to the density of the data to be communicated in the vector space, in order to make the most suitable cuts. To improve the operation of the invention in the embodiment presented in FIG. 1, it is preferable that the indexing function F and the data structure DS are known both by the communication manager 100 and by the reception manager 200. One way of proceeding is for example, for the manager 100, to transmit, in a preamble to the transmission which can be found in a packet header or in a datagram intended for this role, references to an indexing function F and / or a data structure DS, references which are known by the manager 200.

[0197] In other embodiments, the communication manager 100 may transmit the DS data structure itself in a prior exchange. In this case, the size of the DS data structure (and therefore the number of vector data present in the DS data structure, performing the quantification of the vector space of the vector data to be communicated) must be taken into account in evaluating the performance of the methods according to the invention. A prior transmission of a DS data structure adapted to the vector data to be communicated subsequently must be made profitable by numerous communications of vector data.

[0198] Embodiments of the invention include the negotiation of parameter values ​​useful for the transmission of vector data for the invention. Thanks to these embodiments, more advanced possibilities of communication between the communication manager 100 and the reception manager 200 are implemented. We have seen previously that values ​​useful for the execution of the method could be transmitted, for example in a packet header, and make it possible to find elements already defined, such as the data structure DS used in reception to find the vector data SD' close to the data to be communicated SD. But this previous mode is not sufficient when, for example, the data structure DS and / or the indexing function F are learned by automatic learning. The receiver cannot in this case know a priori the elements useful for the operation of the method.

[0199] In this embodiment, a negotiation phase prior to the transmission of the index(es) makes it possible to agree on all the parameters useful for the communication of vector data according to the method. This mode can be combined with the previous one: for example, a first transmission can give the references to a DS data structure, then a negotiation allows the transmitter and the receiver to agree on another parameter such as the size of the packets exchanged.

[0200] Figure 2, for its part, illustrates an example of steps implemented within the framework of an embodiment of the invention.

[0201] In this example, the managers 100 and 200 carry out a negotiation phase making it possible to set up the transfer of vector data from the communication manager 100 to the reception manager 200 through a CNL channel not shown in FIG. 2. For this, the managers 100 and 200 exchange several messages transmitted through the same CNL channel or by another means.

[0202] The first REQ message is sent by the manager 100 to the manager 200 to indicate its intention to communicate vector data and to propose parameters for performing its data communications as best as possible. Such parameters may indicate the format of the vector data to be communicated, which may correspond to a known model name. Another parameter may indicate the intention to fragment the data as well as the size of the fragments obtained after fragmentation. Another parameter may be a reference to the data structure that will be used to quantify the fragments, i.e., the dictionary that groups values ​​associated with indexes. This dictionary may be known to both the manager 100 and the manager 200 and already present in memory. It may, for example, be known as performing good quantification for fragments of a given size for vector data of a given format.The communication manager 100 may also indicate its desire to transmit a dictionary not known by the reception manager 200 but particularly suited to the future communication. The REQ message may also include references to the indexing function F which will be used in embodiments which, again, may be a function already known or a function learned for the purposes of vector data communication. Another useful parameter to be transmitted is the length of the indexes which will be transmitted in place of the fragment values ​​which will be known elsewhere thanks to the data structure. Any other type of parameter useful for the communication may be proposed in the REQ message sent by the communication manager 100 to the reception manager 200.

[0203] The manager 200 may respond with an ANS message that indicates the parameter values ​​it is willing to use for communication. For example, if the communication manager 100 proposes a data structure suitable for a certain fragment size, but the receiving manager 200 does not have this dictionary in memory, it may propose another fragmentation corresponding to a dictionary it has. Several exchanges of REQ-ANS messages may take place between the managers 100 and 200, forming a negotiation of the parameters useful for the communication of vector data between them.

[0204] The communication manager 100 closes the actual negotiation phase by sending a SET message which transmits the actual values ​​on which the managers 100 and 200 have just agreed. For example, the contents of the dictionary may be transmitted here whereas previously the managers 100 and 200 negotiated the size of the dictionary that they were going to use.

[0205] An ACK acknowledgment message may be sent by the reception manager 200 to the communication manager 100 to indicate that the manager 200 has received all the values ​​useful for the communication of vector data and is ready to receive indexes transmitted by the communication manager 100. This exchange of messages corresponds to a possible embodiment of a phase of negotiation of the parameter values ​​useful for the communication of vector data according to the invention, but other embodiments are possible. In other embodiments, the parameter values ​​useful for the communication are not negotiated between the managers 100 and 200 but are included in a datagram forming part of the index transmission.This mode can be used for communications of vector data expressed in standard formats for which data structures carrying out appropriate quantifications are known and maintained in memory by all of the managers 100 and 200 carrying out the methods according to the invention.

[0206] The communication manager 100 begins the communication by performing a FRAG fragmentation of the vector data to be communicated. This operation consists of dividing the vector data to be communicated into several fragments according to a given size which may have been negotiated or which corresponds to a known communication mode for a given vector data format. For example, a vector data item expressed as a 4096-dimensional vector can be divided into 512 fragments of dimension 8. The subsequent operations will apply to each of these fragments individually. The advantage of such a division is, for example, to reduce the size of the data structure to be transmitted or saved in memory, since the elements present in this data structure will no longer be vector data of dimension 4096 but fragments of dimension 8.

[0207] The communication manager 100 continues, in this exemplary embodiment, with a quantification operation QNT. This operation may consist of searching in the data structure for the closest fragment according to a given distance from a fragment of the data to be communicated. The indexing operation IND then consists of finding in the data structure the index corresponding to this closest fragment. In other embodiments, an index is calculated directly for the fragment of the data to be communicated and the quantification operation QNT corresponds to the construction of the data structure used by the managers 100 and 200.

[0208] The communication manager 100 continues with a transmission operation TRS which comprises sending a message comprising the index I found in the data structure which has been negotiated, index I corresponding to a fragment present in the data structure close, according to the given distance, to the fragment of the data to be communicated. These operations of quantification QNT, indexing IND and transmission TRS are repeated for all the fragments corresponding to the vector data to be communicated which were obtained by the fragmentation operation FRG.

[0209] The reception manager 200 will then carry out a succession of RCP reception operations of transmitted index I and RDG reading in the data structure of fragments corresponding to the received indexes I. A DFRG defragmentation operation then makes it possible to reconstruct the vector data to be communicated from the read fragments which correspond to the received indexes. This reconstructed vector data will be that which will be used in artificial intelligence applications in place of the vector data to be communicated. The QNT quantization and / or IND indexing operations ensure that the indexes obtained are chosen according to a distance measured between the data to be communicated and that reconstructed from the fragments corresponding to the received indexes. Figure 3, for its part, illustrates examples of possible distortions undergone by vector data during its communication and reception.

[0210] In the example of Figure 3, to better visualize the distortions introduced by the communication and reception management methods according to the invention, it is assumed that the vector data considered are vectors in a 3-dimensional space represented by the X, Y, Z reference frame located at the origin of these vectors. This example is not realistic because the vector data or fragments that will be communicated will often be of a larger dimension, and which is a power of two, but the principles remain the same in the vector space concerned.

[0211] The goal is to communicate the SD vector data.

[0212] It is assumed that, in this example, the method uses an indexing function F and that the indexing function F directly calculates an index I from SD, but that this index I is not present in the data structure DS. In certain embodiments, the method will not directly transmit the index I but will look for a close index I', which is indeed associated with a vector data item SD' in the data structure DS. In other embodiments, the index I calculated by the function F will be considered as the index obtained and directly transmitted. The index I' is transmitted in the CNL channel but this introduces transmission errors. The index received is then the index I”, which is assumed here not to be present in the data structure DS. The method according to the invention will then look for an index I'” close to the index I” such that I'” is present in the data structure DS.The vector data SD'” associated with the index I'” in the data structure DS will then be used in various applications instead of the initial data SD. The accumulation of distortions means that the data used SD'” is not the initial data SD, but these distortions are reduced to a minimum by suitable choices of indexes based on a distance D measured between vector data, the indexing function F, the function calculating the distance D measured between vector data, and the quantification by the data structure DS of the vector space XYZ in which the vector data are located.

[0213] In the previous example, the indices I and I” are not present in the DS data structure. In some embodiments, the DS data structure includes an index-vector data association for all possible values ​​of the indices. In this case, the index I directly calculated from the SD data will be transmitted directly by the CNL channel. And the vector data that will be used finally will be directly the one associated with the received index I” after the transmission errors of the CNL channel have transformed the index I into I”.

[0214] Furthermore, it is important to understand that, even if all possible index values ​​are present in the data structure DS, the vector data associated with an index calculated in the data structure DS will not generally be the vector data for which the index was calculated. Indeed, there is a finite number of possible index values ​​whereas vector data, which are vectors in a vector space of real numbers, can take an infinite number of values. A calculated index I will therefore be the same for an entire set of vector data whereas only one vector data will be associated with this index in the data structure DS. To improve the efficiency of the method according to the invention, the vector data which are sent to the same index by the indexing function F must be close to each other according to the distance D measured between them. These data close to each other then form a distinct zone in the vector space.The vector data that will be chosen as being associated with the index calculated for the vector data of this zone can be for example the data located at the barycenter of this zone to improve the representativeness of the vector data which have the same calculated index associated with a single vector data.

[0215] Figure 4, for its part, illustrates experimental results obtained during the development of the methods according to the invention.

[0216] In this experiment, we wanted to show to what extent the distortions introduced by the methods for managing the communication and reception of vector data according to the invention impacted the artificial intelligence tasks in which the communicated vector data are used.

[0217] The Hugging Face research community (accessible via the website http: / / huggingface.co) offers examples of tasks, models, and datasets for comparing algorithms in the field of artificial intelligence. For our experiments, we considered the banking77 dataset (accessible at https: / / huggingface.co / datasets / banking77). This dataset corresponds to several tens of thousands of sentences (in English) corresponding to 77 possible requests in the banking field, such as requesting a block in the event of a stolen bank card, or requesting an account balance, or other examples.

[0218] We converted this textual data set into vector data according to the Mistral-7B model with a size of 4096 reals encoded in float32. The vector data therefore represents a semantic content which is that of the initial texts. These semantic contents associated with the vector data must be preserved as much as possible despite the distortions introduced by the methods according to the invention allowing vector data to be communicated.

[0219] We took a portion of this converted vector data (and the known classification of the associated semantic content) to train a neural network to classify a vector data item into one of the 77 categories of requests in the banking domain. This is therefore a classic classification task of artificial intelligence application, and which uses vector data for its manipulations. The performance of the trained neural network, measured on vector data other than those used for training the neural network, is presented in Figure 3 by the dashed line, labeled nominal model, near the top. They are greater than 90% correct classification. Conversely, random performance corresponds to a success of 1 / 77. It is represented by the dotted line, labeled random model, close to zero (indeed, 1 / 77 = 1.3%).

[0220] Between these two extremes, we find three curves that represent the results of classification tasks that are obtained after using three distinct ways of communicating vector data with the methods according to the invention in the presence of a CNL channel introducing more and more disturbances.

[0221] To obtain the curves in question, we simulated the communication of vector data between two devices EQ1, EQ2 in a CNL channel and submitted the communicated data to the neural network that we trained. The difference with the nominal model line therefore shows the impact of the communication of vector data according to the method on the classification task carried out by the neural network. The CNL channel is simulated as having a variable error rate in the transmission of bits, a rate which is called BER (acronym for bit error ratio). The horizontal scale is logarithmic and shows simulations for a CNL channel having an error rate that varies from 10 -3 , that is, a one in 1000 chance of a one-bit transmission error up to more than 10 -1 , that is, more than a one in ten chance of transmission error, passing through 10 -2 (one in 100 chance of transmission error).

[0222] In the three simulated communication examples, we chose to fragment the vector data into fragments of dimension 8. A vector data which initially comprises 4096 reals will therefore be divided into 512 fragments which comprise 8 reals encoded in float32, i.e. 512 fragments of 8 * 32 = 256 bits.

[0223] We then performed quantifications suitable for 8-dimensional vector data fragments. For this, we took another corpus proposed by the Hugging Face research community, namely the Locutusque UltraTextbooks 2.0 corpus, accessible at the link http: / / huggingface.co / datasets / Locutusque / UltraTextbooks-2.0, and performed training using a clustering algorithm. It should be noted that this corpus of textual data has nothing to do with the banking domain. The quantifications learned here could therefore be used to transfer vector data representing texts outside the banking domain in a probably equally efficient way.

[0224] We have carried out three quantifications from this corpus to learn 2, then 16, then 1024 vector data which are used to quantify fragments of dimension 8. The 2 fragments of quantized vector data can then be represented by an index of length 1 bit; the 16 fragments of vector data can be represented by an index of length 4 bits; and the 1024 fragments of vector data by an index of length 10 bits. An indexing function is then used to calculate an index of length 1, 4 or 10 bits. The calculated indexes are transferred and subjected to simulated variable error rates. The fragments corresponding to the received indexes, having undergone transmission errors, are recovered, then a complete vector data is constructed from the fragments and is submitted to the trained neural network, and the classification performance is measured.The classification performances are of course between the nominal model line (classification without transmission of semantic data) and the random model line (random classification).

[0225] We see that, for indexes of length 4 or 10 bits, the classification performances are very close to the nominal performance up to an error rate of 10 -2 . However, the use of these indexes corresponds to a reduction in the size of the transmitted data from 8 times 32 = 216 bits to 4 or 10 bits. We therefore have here a reduction factor of at least 20 and up to 54 for the 4-bit length index if we had chosen to communicate the vector data by directly transmitting the bits which represent the 512 fragments of 8 real numbers encoded on 32 bits. This very strong reduction is achieved with a minimal loss at the level of the classification task. The performances remain above 80% up to a fairly high error rate of 10 -1 .

[0226] 1-bit quantization has a maximum performance of just under 80%. These results are therefore not very good, but not negligible either given a reduction factor of 216 in the size of the transmitted data.

[0227] Eventually, as the BER increases to approach a rate of one in two errors, classification performance drops to that of random classification.

[0228] These experiments therefore clearly show the interest of the invention which offers a reduction in the number of bits transmitted compared to a direct transmission of the bits forming the vector data communicated by a very appreciable factor and a slight drop in classification performance, even in the presence of transmission errors at a significant rate.

[0229] Finally, it should be noted here that, in this text, the term "module" can correspond to a software component as well as to a hardware component or a set of hardware and software components, a software component itself corresponding to one or more computer programs or sub-programs or, more generally, to any element of a program capable of implementing a function or a set of functions as described for the modules concerned. In the same way, a hardware component corresponds to any element of a hardware assembly capable of implementing a function or a set of functions for the module concerned (integrated circuit, smart card, memory card, etc.).

Claims

Claims 1. Method for managing the communication of vector data (SD), called data to be communicated, from a first device (EQ1) to a second device (EQ2), the method comprising: - obtaining an index (I') by applying an indexing function (F) to the data to be communicated (SD), the indexing function (F) calculating an index from a vector data item, the indexing function (F) being such that, the closer two vector data items are according to a distance (D) measured between them, the closer the indexes calculated for these two vector data items by the function (F) are between them, said index (I') making it possible to find in a data structure (DS) another vector data item (SD'), called communicated data item, the data structure (DS) associating indexes with vector data, the obtained index (I') being able to be transmitted from the first equipment item (EQ1) to the second equipment item (EQ2).

2. Management method according to claim 1 characterized in that the distance (D) measured between the data to be communicated (SD) and the communicated data (SD') is a function of the proximity between the semantic contents associated with the two vector data (SD, SD').

3. Management method according to one of claims 1 or 2 characterized in that the index (I') is obtained by searching in the data structure (DS) for the vector data (SD') closest to the data to be transmitted (SD) according to the measured distance (D), and the transmitted index is the index (I') associated with said vector data (SD') in the data structure (DS).

4. Management method according to one of claims 1 to 3 characterized in that the distance (D) between vector data is measured using cosine similarity.

5. Management method according to one of claims 1 to 4 characterized in that the indexing function (F) is a locality-sensitive hash function.

6. Management method according to one of claims 1 to 5, characterized in that the method comprises a prior step of fragmenting the data to be communicated (SD) into several fragments and the subsequent steps of the method apply to said several fragments.

7. Management method according to one of claims 1 to 6, characterized in that the data structure (DS) is defined beforehand according to the format of the data to be communicated (SD).

8. Management method according to one of claims 1 to 7 characterized in that the data structure (DS) is obtained by automatic learning based on the vector data already communicated and / or intended to be communicated.

9. Management method according to claim 8 characterized in that the automatic learning of the data structure (DS) uses the k-means method where k is the number of vector data present in the data structure (DS).

10. Method for managing the reception of vector data (SD') characterized in that it comprises the reception of an index (I'), called received index, followed by the reading of vector data (SD') in a data structure (DS) associating indexes with vector data as a function of the received index (I'). 1 1. Management method according to claim 10 characterized in that the method comprises, if the received index (I') does not correspond to any index present in the data structure (DS), a search in the data structure (DS) for the index closest to the received index (I') and in that the vector data (SD') read in the data structure (DS) is that which is associated with said closest index.

12. Management method according to one of claims 10 or 11 characterized in that the method comprises, if the received index (I') corresponds to a fragment of vector data, the reconstitution of complete vector data from the fragments of vector data corresponding to the received indexes.

13. Communication manager (100) capable of managing the communication of vector data (SD), called data to be communicated, from a first device to a second device, the communication manager comprising: - a module (101) for obtaining an index (I'), the obtaining module being able to obtain an index (I') by applying an indexing function (F) to the data to be communicated (SD), the indexing function (F) calculating an index from a vector data item, the indexing function (F) being such that, the closer two vector data items are according to a distance (D) measured between them, the closer the indexes calculated for these two vector data items by the function (F) are between them, said index (I') making it possible to find in a data structure (DS) another vector data item (SD'), called communicated data, the data structure (DS) associating indexes with vector data, the module (101) for obtaining an index being able to provide an index to an index transmitter able to transmit an index from the first equipment item (EQ1) to the second equipment item (EQ2).

14. Telecommunications equipment (EQ1) including a communications manager (100) capable of managing the communication of vector data (SD) according to claim 13.

15. Reception manager (200) of vector data (SD') comprising: A module (201) for receiving an index (!'), called index received; A reader (202) of a vector data (SD') in a data structure (DS) associating indexes with vector data according to the received index (I').

16. Telecommunications equipment (EQ2) comprising a management entity (200) capable of managing the reception of vector data (SD') according to claim 15.

17. A computer program capable of being implemented by a communications manager (100), the program comprising code instructions which, when executed by a processor, performs the steps of the management method defined in claim 1.

18. A computer program capable of being implemented by a reception manager (200), the program comprising code instructions which, when executed by a processor, performs the steps of the management method defined in claim 10.

Images