DATA COLLECTION SYSTEM ARCHITECTURE

Abstract

A system (100) is configured to collect data from computer platforms accessible via access interfaces (101, 102, 103), and comprises: a metasearch engine (130) operating in a generic query format to order the data collection; a translator (120) transcribing the queries generated by the metasearch engine (130) in the generic format into queries specifically adapted to the access interfaces (101, 102, 103) of the computer platforms to which said queries are intended, the translator (120) being equipped with a plugin (111, 112, 113) specific to each said access interface (101, 102, 103);an identity manager (140) configuring the translator (120) with access parameters specific to the computer platforms to which said requests are addressed, and an orchestrator (150) configured to coordinate the metasearch engine (130) and the identity manager (140), in order to perform data collection from the computer platforms. Figure to be published with the abbreviation: Fig. 1;

Description

Title of the invention: DATA COLLECTION SYSTEM ARCHITECTURE Technical field

[0001] The present invention relates to the field of collecting data available on computer platforms made accessible via one or more communication networks, such as social network hosting platforms. STATE OF PRIOR ART

[0002] The collection of data available on computer platforms made accessible via one or more communication networks, and more particularly the collection of data on social networks, is a complex subject due to the multitude of platforms concerned, each having specificities and automatic access restrictions to the data which are constantly evolving.

[0003] As a result, several data collection platforms have emerged in recent years, but these differ in scope (supported platforms, access to the deep web, also called the invisible web, access to the dark web, etc.), as well as in the capacity to pre-process the collected data (optical character recognition, reverse image search, machine translation, automatic analysis of emotions, extraction of named entities, etc.), which leads to access interfaces with extremely diverse and changing functionalities.

[0004] From the point of view of an analyst wishing to benefit from the capabilities of the different platforms, the automatic and unified management of each data source poses a problem of designing a data collection system architecture (tool) that is easily adaptable to the changing nature of the data source platforms.

[0005] It is then desirable to overcome at least this drawback of the prior art, and to do so in a simple, effective and inexpensive manner. It is particularly desirable to provide a data collection solution that is flexible and easily adaptable in the event of changes in functionality and access to data source platforms. Statement of the invention

[0006] To this end, a data collection system is proposed for collecting data from computer platforms accessible via access interfaces, the data collection system comprising: a metasearch engine configured to operate under a generic query format in order to order the collection of data; a translator configured to transcribe the queries generated by the meta-search engine in the generic format into queries specifically adapted to the access interfaces of the IT platforms for which said queries are intended, the translator being equipped with a plugin specific to each said access interface; an identity manager configured to configure the translator with access parameters specific to the IT platforms for which said queries are intended; and an orchestrator configured to coordinate the meta-search engine and the identity manager, in order to carry out the data collection from the IT platforms. Thus, particularly thanks to the implementation of the translator and the plugins, the data collection is flexible and easily adaptable in the event of changes in functionalities and access to the IT platforms (eg, social networks), which are sources of data.

[0007] In a particular embodiment, the metasearch engine comprises three sub-metasearch engines: a first sub-metasearch engine dedicated to regular monitoring schedules by data collection; a second sub-metasearch engine dedicated to single-purpose queries, in order to enable immediate collection of data on demand; and a third sub-metasearch engine dedicated to one-off investigation by iterative searches.

[0008] In a particular embodiment, the first sub-metasearch engine is configured to use a descriptor of entities to be monitored and a filtering descriptor comprising filter characteristics.

[0009] In a particular embodiment, the first sub-metasearch engine and / or the third sub-metasearch engine are configured to use a constraint descriptor comprising identity constraint information and / or priority constraints.

[0010] In a particular embodiment, the second sub-metasearch engine and / or the third sub-metasearch engine are configured to use a resource-related rule descriptor serving as a reference for data collection, and an entity-related rule descriptor serving as a reference for data collection.

[0011] In a particular embodiment, the identity manager is configured to use a credential information descriptor, a proxy information descriptor, and a behavior and artifacts descriptor.

[0012] In a particular embodiment, the metasearch engine is configured to operate under a generic response format in order to process the data collection, and the translator is configured to transcribe responses received via the access interfaces into responses in the generic format.

[0013] In a particular embodiment, at least one search engine is im implemented in at least one respective plugin to perform research and data collection from the computing platform to which said plugin specifically corresponds.

[0014] A data collection method is also proposed for carrying out data collection from computer platforms accessible via access interfaces, the method being implemented by a data collection system, the method comprising: generating, by a metasearch engine of the data collection system, in a generic format, queries in order to order the data collection; transcribing, by a translator of the data collection system, the queries generated by the metasearch engine in the generic format into queries specifically adapted to the access interfaces of the computer platforms for which said queries are intended, the translator being equipped with a plugin specific to each said access interface;configure the translator, by an identity manager of the data collection system, with access parameters specific to the IT platforms to which said requests are intended, and coordinate the metasearch engine and the identity manager, by an orchestrator of the data collection system, in order to carry out the data collection from the IT platforms.;

[0015] Also provided is a computer program product comprising instructions for implementing the method presented above, when said instructions are executed by a processor. Also provided is a (non-transitory) information storage medium storing a computer program comprising instructions for implementing the method presented above, when said instructions are read from the information storage medium and executed by a processor. Brief description of the drawings

[0016] The following description of at least one embodiment is set forth in relation to the accompanying drawings, among which:

[0017] [Fig-1] schematically illustrates a data collection system architecture;

[0018] [Fig.2] schematically illustrates an example of a hardware unit allowing the implementation of the data collection system architecture of [Fig.l];

[0019] [Fig.3] schematically illustrates a particular metasearch engine arrangement of the data collection system architecture of [Fig.l]; and

[0020] [Fig.4] schematically illustrates a flowchart of a data collection algorithm implemented by the data collection system architecture of [Fig.l].

[0021] DETAILED DESCRIPTION OF EMBODIMENTS

[0022] [Fig.l] schematically illustrates an architecture of a data collection system 100. Such a data collection system 100 is particularly adapted to the field of information monitoring.

[0023] Data collection is carried out from computer platforms made accessible via one or more communication networks, such as social network hosting platforms or databases. Exchanges with these computer platforms are carried out using access interfaces, called APIs (Application Programming Interfaces), which allow access to these computer platforms and requests to be made to these computer platforms in order to collect data available there. These access interfaces form an abstraction layer of the internal functions or services provided by said computer platforms, by exposing in a predefined manner the functionalities offered by these services. Thus, the manner of accessing these computer platforms (eg, combination of user ID and password) and their services (eg, social networks), as well as the format of the resulting requests and responses, is predefined by these access interfaces. For purely illustrative purposes, three access interfaces API P#1 101, API P#2 102 and API P#3 103 are shown in [Fig.l].

[0024] To adapt to these access interfaces, the data collection system 100 further comprises plugins adapted to said access interfaces. The data collection system 100 comprises such a plugin for each access interface. Thus, in [Fig.l], the data collection system 100 comprises three plugins P_I P#1 111, P_I P#2 112 and P_I P#3 113 for respectively the three access interfaces API P#1 101, API P#2 102 and API P#3 103.

[0025] The data collection system 100 further comprises a translator TR 120. The translator TR 120 is equipped with said plugins P_I P#1 111, P_I P#2 112 and P_I P#3 113 and transcribes data collection requests, in a generic format, into requests specifically adapted to the access interface of the computer platform for which said data collection requests are intended. In addition, the translator TR 120 transcribes responses to said requests, which are therefore received in a format specific to said plugin, into responses in a generic format. Thus, thanks to the transducer TR 120, the requests and the corresponding responses are respectively generated and used in the same format within the data collection system 100.The above-mentioned transcription operations are joint operations of the translator TR 120 and the plugins P_I P#1 111, P_I P#2 112 and P_I P#3 113, the translator TR 120 providing the general transcription framework and each of the plugins P_I P#1 111, P_I P#2 112 and P_I P#3 113 providing the extension necessary for adaptation to the IT platform (eg, social network) to which the plugin in question is dedicated.

[0026] In order to generate these requests, and to process the corresponding responses, the data collection system 100 comprises a metasearch engine SME 130. An advantageous embodiment of the metasearch engine SME 130 is detailed below in relation to [Fig. 3]. A metasearch engine is a manager common to several search engines and therefore implements a function of interfacing and centralizing searches and data collection carried out on the different IT platforms (eg, social networks). In the present case, the search engines are preferably part of the functions, implemented by the IT platforms (eg, social networks), which are exported and made accessible by the access interfaces API P#1 101, API P#2 102 and API P#3 103.Alternatively, one or more search engines may be implemented directly in the data collection system 100, for example to provide more advanced functionalities than the search engine that would be provided by the computing platform (eg, social network) in question or to exploit a more efficient data search algorithm. In this variant, each search engine that is implemented directly in the data collection system 100 is included in the plugin P_I P#1 111, P_I P#2 112 or P_I P#3 113 corresponding to the computing platform (eg, social network) for which the search engine in question operates.

[0027] The SME metasearch engine 130 therefore operates under the generic query and response format mentioned above, in order to order the collection of data. The translator TR 120 and the plugins P_I P#1 111, P_I P#2 112 and P_I P#3 113 thus allow the SME metasearch engine 130 to operate independently of the computer platforms from which the data is collected.

[0028] The data collection system 100 further comprises an identity manager IDM 140. The identity manager IDM 140 is responsible for configuring the translator TR 120 with access parameters specific to the IT platform (e.g., social network) to which a data collection request must be transmitted. Thus, thanks to this configuration, the IT platform in question is accessed using a chosen identity (e.g., login and password) in accordance with the access constraints imposed by the IT platform (e.g., social network) in question.

[0029] The identity manager IDM 140 preferentially uses, to do this, a CRED 161 descriptor of identification information (“credentials” in English) comprising, for each computer platform (e.g., social network), an association of at least one connection identifier to said computer platform, a connection password corresponding to said connection identifier and an identifier of said computer platform sufficient to access said computer platform, such as a URL (“Uniform Resource Locator” in English) of a social network, of a blog, website... .

[0030] The identity manager IDM 140 also preferentially uses a PXY 162 proxy information descriptor comprising, for each computer platform (e.g., social network), if necessary to access said computer platform, at least one location information of a proxy device, a type and / or identifier of the communication network on which said proxy device is located (LTE network (“Long-Term Evolution” in English), FFTH (“Fiber-To-The-Home” in English)...) and identifiers of the IP address (“Internet Protocol” in English) and port number of said proxy device.

[0031] The identity manager IDM 140 also preferentially uses a behavior and artifact descriptor B&A 163 comprising, for each computer platform (e.g., social network), at least one piece of application context information (browser, dedicated application, etc. with, if necessary, an associated version number) to be used to connect to said computer platform, information on the time and / or duration of connection (if the platform imposes limitations of this order) and information on the subject or theme concerned by the data collection, such as, for example, a set of keywords representative of said subject or theme.

[0032] The descriptors mentioned above can be produced in the form of files, according to a predefined editing format, thus making it possible to easily modify the information relating to the IT platforms (e.g., social networks) useful to the data collection system 100 for carrying out data collection from said IT platforms.

[0033] The data collection system 100 further comprises an orchestrator ORCH 150. The orchestrator ORCH 150 is responsible for coordinating the operations of the meta-search engine SME 130 and the identity manager IDM 140, to respond to data collection requests emanating from an external analysis system, typically via a communication interface, or from a human operator, typically via a human-machine interface (not shown in [Fig.l]) of the data collection system 100.

[0034] Thus, when a new IT platform (eg, access to a new social network) must be supported, the data collection system 100 can be easily adapted by adding a new plugin, dedicated to the access interface corresponding to this new IT platform, to the translator TR 120. No change is necessary within the orchestrator ORCH 150, nor the metasearch engine SME 130, nor the identity manager IDM 140. Similarly, when an evolution of the access interface (eg, change of access method to functionalities) is necessary, only the plugin dedicated to the access interface corresponding to the IT platform concerned must be modified. Only new parameters may have to be be added to the descriptors used by the identity manager IDM 140, to take into account this new IT platform or changes in access accounts to an IT platform (eg, social network user account) or access conditions (eg, declaration of a proxy device) to an IT platform. The data collection system architecture 100 presented in [Fig.l] is therefore flexible and easily adaptable.

[0035] [Fig.2] schematically illustrates an example of a hardware unit 200 adapted to implement the data collection system 100.

[0036] The hardware unit 200 comprises, connected by a communication bus 210: a processor or CPU (for “Central Processing Unit” in English) 201, or a cluster of such processors, such as for example GPUs (“Graphics Processing Units” in English); a RAM (for “Random Access Memory” in English) 202; a ROM (for “Read Only Memory” in English) 203, or a rewritable memory of the EEPROM type (“Electrically Erasable Programmable ROM” in English), for example of the Flash type; a data storage device, such as a hard disk HDD (for “Hard Disk Drive” in English) 204, or a storage media reader, such as an SD (for “Secure Digital” in English) card reader;a set of input and / or output interfaces, such as communication interfaces 205, allowing in particular the data collection system 100 to interact with other equipment and in particular with the computer platforms from which the data collections are carried out.;

[0037] The processor 201 is capable of executing instructions loaded into the RAM 202 from the ROM 203, from an external memory (not shown), from a storage medium, such as an SD card or the HDD 204, or from a communication network. When the hardware unit 200 is powered on, the processor 201 is capable of reading instructions from the RAM 202 and executing them. These instructions form a computer program causing the processor 201 to implement the architecture and methods described herein.

[0038] All or part of the steps, behaviors and algorithms described here can thus be implemented in software form by executing a set of instructions by a programmable machine, such as a DSP (Digital Signal Processor) or a processor, or be implemented in hardware form by a machine or a dedicated component (chip) or a dedicated set of components (chipset), such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit).

[0039] Generally speaking, the data collection system 100 therefore comprises electronic circuitry arranged and configured to implement the steps, behaviors and algorithms described herein.

[0040] [Fig.3] schematically illustrates the SME metasearch engine 130, in a particular embodiment.

[0041] The metasearch engine SME 130 then comprises a first sub-metasearch engine MNTOR 131 dedicated to regular monitoring programs by data collection. Regular monitoring is associated with a storage of the history of the queries carried out as part of said regular monitoring, as well as the data collected, thus making it possible to determine or detect a possible change in the data collected. The storage of the queries makes it possible to request the IT platform concerned without prejudging its historicization capabilities.

[0042] The first sub-metasearch engine MNTOR 131 preferentially uses, to do this, an ENT 311 descriptor of entities to be monitored, such as particular users, particular groups, etc.

[0043] The first MNTOR 131 sub-metasearch engine also preferentially uses a filtering descriptor FLTR 312 comprising filter characteristics, such as keywords, selections of computer platforms (eg, social networks) to target, time slots for data collection, etc.

[0044] The first sub-metasearch engine MNTOR 131 preferentially also uses a descriptor CON 313 of constraints comprising information on identity constraints and / or priority constraints.

[0045] The descriptors mentioned above can be produced in the form of files, according to a predefined editing format, thus making it easy to modify the information relating to regular monitoring by the data collection system 100.

[0046] The metasearch engine SME 130 also includes a second sub-metasearch engine SQ 132 dedicated to single query queries, in order to enable immediate collection of data on demand.

[0047] The metasearch engine SME 130 also includes a third sub-metasearch engine INV 133 dedicated to a one-off investigation by iterative searches. This is an approach halfway between the operation of the first sub-metasearch engine MNTOR 131 and the second sub-metasearch engine SQ 132. Whereas the regular monitoring carried out by the first sub-metasearch engine MNTOR 131 concerns a broad spectrum of data but a low search depth, the one-off investigation concerns a more restricted spectrum of data (for example, around an entity (person) detected during the regular monitoring and / or around an image previously collected during the regular monitoring) but with a greater search depth (i.e., accumulation of several successive searches by defining, for a new search iteration, a pivot around data collected during a previous research iteration).

[0048] The third sub-metasearch engine INV 133 preferentially uses rule descriptors which constitute entry points for the data collection to be carried out for the specific investigation, thus making it possible to reduce the search spectrum. Each rule descriptor corresponds to a specific search methodology, for example a data search methodology from an image and a search methodology from a person's identity. These rule descriptors can also be used by the second sub-metasearch engine SQ 132.

[0049] The second sub-metasearch engine SQ 132 and / or the third sub-metasearch engine INV 133 use, for example, to do this, a descriptor R_R 321 of rules relating to resources serving as a reference for the collection of data, such as for example an image around which the investigation is to be carried out (i.e., collection of data relating to this image).

[0050] The second sub-metasearch engine SQ 132 and / or the third sub-metasearch engine INV 133 use, for example, to do this, a descriptor E_R 322 of rules relating to entities serving as a reference for the collection of data, such as for example an identity of a person around which the investigation is to be carried out (i.e., collection of data relating to this person).

[0051] The third sub-metasearch engine INV 133 can also use the constraint descriptor CON 313 mentioned above in relation to the first sub-metasearch engine MNTOR 131.

[0052] The descriptors mentioned above can be produced in the form of files, according to a predefined editing format, thus making it easy to modify the information relating to the specific investigations carried out by the data collection system 100.

[0053] [Fig.4] schematically illustrates a flowchart of a data collection algorithm implemented by the data collection system 100.

[0054] In a step 401, the data collection system 100 is configured to meet a data collection need, typically to carry out information monitoring. For example, the data collection system 100 is configured to carry out regular monitoring programming by data collection, or to carry out a single-purpose data collection request, or to carry out a one-off investigation. The ORCH orchestrator 150 coordinates the metasearch engine SME 130 and the identity manager IDM 140, in order to carry out the data collection from the IT platforms (e.g., social networks). This triggers a generation, in a step 402 by the metasearch engine SME 130, of generic queries adapted to the data collection to be carried out, as well as a pa setting up of the translator TRAD 120, in a step 403 by the identity manager IDM 140, with access parameters specific to the IT platforms (eg, social networks) to which the data collection requests are addressed.

[0055] In a step 404, the translator TRAD 120, using each plugin P_I P#1 111, P_I P#2 112 and P_I P#3 113 dedicated to a computer platform to which at least one data collection request must be addressed, translates the generic requests generated by the metasearch engine SME 130 into requests specifically adapted to the access interfaces API P#1 101, API P#2 102 and API P#3 103 of the computer platforms to which said requests are intended.

[0056] In a step 405, the data collection system 100 performs the data collection by transmitting said requests specifically adapted to the access interfaces API P#1 101, API P#2 102 and API P#3 103, and receives corresponding responses which include the collected data.

[0057] In a step 406, the translator TRAD 120, using each graft P_I P#1 111, P_I P#2 112 and P_I P#3 113 dedicated to a computer platform to which at least one data collection request has been addressed, preferentially translates the responses received into generic responses usable by the data collection system 100.

Claims

1. Claims A data collection system (100) for performing data collection from computing platforms accessible via access interfaces (101, 102, 103), the data collection system (100) comprising: - a metasearch engine (130) configured to operate under a generic query format in order to order the collection of data; - a translator configured to transcribe the queries generated by the metasearch engine (130) in the generic format into queries specifically adapted to the access interfaces (101, 102, 103) of the computer platforms for which said queries are intended, the translator (120) being equipped with a plugin (111, 112, 113) specific to each said access interface (101, 102, 103); - an identity manager (140) configured to configure the translator (120) with access parameters specific to the computing platforms to which said requests are intended; and - an orchestrator (150) configured to coordinate the metasearch engine (130) and the identity manager (140), in order to carry out the collection of data from the computing platforms, the data collection system (100) being such that the metasearch engine (130) comprises three sub-metasearch engines: - a first sub-metasearch engine (131) dedicated to regular monitoring schedules by data collection; - a second sub-metasearch engine (132) dedicated to single-purpose queries, in order to enable immediate collection of data on demand; and - a third sub-metasearch engine (133) dedicated to a one-off investigation by iterative searches; the data collection system (100) being further such that: - the first sub-metasearch engine (131) is configured to use a descriptor (311) of entities to be monitored, and a filtering descriptor (312) comprising filter characteristics; - the first sub-metasearch engine (131) and / or the third sub-metasearch engine (133) are configured to use a constraint descriptor (313) comprising identity constraint information and / or priority constraints; and - the second sub-metasearch engine (132) and / or the third sub-metasearch engine (133) are configured to use a resource rule descriptor (321) for reference to data collection, and an entity rule descriptor (322) for reference to data collection.

2. The data collection system (100) of claim 1, wherein the identity manager (140) is configured to use an identification information descriptor (161), a proxy information descriptor (162), and a behavior and artifacts descriptor (163).

3. The data collection system (100) of claim 1 or 2, wherein the metasearch engine (130) is configured to operate under a generic response format to process the data collection, and the translator (120) is configured to transcribe responses received via the access interfaces (101, 102, 103) into responses in the generic format.

4. A data collection system (100) according to any one of claims 1 to 3, wherein at least one search engine is implemented in at least one respective plugin for performing search and data collection (111, 112, 113) from the computing platform to which said plugin (111, 112, 113) specifically corresponds.

5. Data collection method intended to carry out data collection from computer platforms accessible via access interfaces (101, 102, 103), the method being implemented by a data collection system (100), the method comprising: - generating (402), by a metasearch engine (130) of the data collection system (100), in a generic format, queries in order to order the data collection; - transcribing (404), by a translator (120) of the data collection system (100), the queries generated by the metasearch engine (130) in the generic format into queries specifically adapted to the access interfaces (101, 102, 103) of the computer platforms for which said queries are intended, the translator (120) being equipped with a plugin (111, 112, 113) specific to each said access interface (101, 102, 103);- configuring the translator (120), by an identity manager (140) of the data collection system (100), with access parameters specific to the computer platforms to which said requests are; destinies; and - coordinate the metasearch engine (130) and the identity manager (140), by an orchestrator (150) of the data collection system (100), in order to carry out the collection of data from the IT platforms; the data collection system (100) being such that the metasearch engine (130) comprises three sub-metasearch engines: - a first sub-metasearch engine (131) dedicated to regular monitoring programs by data collection; - a second sub-metasearch engine (132) dedicated to single-purpose queries, in order to enable immediate collection of data on demand; and - a third sub-metasearch engine (133) dedicated to a one-off investigation by iterative searches; the process being further such that: - the first sub-metasearch engine (131) uses a descriptor (311) of entities to be monitored, and a filtering descriptor (312) comprising filter characteristics; - the first sub-metasearch engine (131) and / or the third sub-metasearch engine (133) use a constraint descriptor (313) comprising identity constraint information and / or priority constraints; and - the second sub-metasearch engine (132) and / or the third sub-metasearch engine (133) use a descriptor (321) of rules relating to resources serving as a reference for the data collection, and a descriptor (322) of rules relating to entities serving as a reference for the data collection.

6. A computer program product comprising instructions for implementing the method according to claim 5, when said instructions are executed by a processor.

7. An information storage medium storing a computer program comprising instructions for implementing the method according to claim 5, when said instructions are read from the information storage medium and executed by a processor.

Images