An information collection and analysis method, device, equipment, medium and product

Abstract

The present invention discloses a method, apparatus, device, medium and product for information collection and analysis, relating to the technical field of cloud computing. Through a gateway with multi-protocol recognition function and a dynamic traffic processing platform, this solution monitors the language model call requests transmitted by multiple data sources, realizes the unified management of LLM call data, and adapts to multiple protocols. When receiving a language model call request, it forwards the language model call request to the language model service cluster to generate corresponding Q&A data, and obtains the user behavior data generated by the target data source based on the Q&A data, so as to perform full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data. The analysis process not only focuses on call requests and responses, but also takes into account the corresponding user behavior, making the generated call analysis results more accurate and improving the credibility of the application effect analysis of the LLM.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and particularly to a method, device, equipment, medium and product for information collection and analysis. Background Art

[0002] With the rapid progress of artificial intelligence technology, large language models (LLMs) are increasingly widely used inside and outside enterprises, covering multiple fields such as integrated development environments (IDEs), user interfaces (UIs), and application programming interface (API) calls. In order to better evaluate and optimize the performance of LLMs, understand user interaction patterns, and perform fine-grained management and resource scheduling, the comprehensive collection and analysis of LLM call information have become crucial.

[0003] However, since the call information from different entrances is scattered in various systems and there is a lack of a unified collection mechanism, it is difficult to centrally manage the data, and cross-entrance analysis becomes complex and difficult. Secondly, existing collection technologies often only focus on requests and responses, while ignoring key information such as user behavior data, such as the adoption of code completion or UI interaction behavior, which limits the in-depth analysis of the application effect of LLMs.

[0004] In view of the above, how to solve the problem that the current LLM call data is scattered and not conducive to management, the collection dimension is single and ignores user behavior data, and it is difficult to achieve comprehensive, unified and efficient collection and analysis is an urgent problem for those skilled in the art in this field. Summary of the Invention

[0005] The present invention provides a method for information collection and analysis to at least solve the problems that the current LLM call data is scattered and not conducive to management, the collection dimension is single and ignores user behavior data, and it is difficult to achieve comprehensive, unified and efficient collection and analysis.

[0006] The present invention provides a method for information collection and analysis, including:

[0007] Monitoring language model call requests transmitted by multiple data sources through a gateway with multi-protocol recognition function and a dynamic traffic processing platform; wherein the data sources at least include user interfaces, application programming interfaces, and integrated development environment plugins;

[0008] When receiving a language model call request transmitted by a target data source, forwarding the language model call request to a language model service cluster to generate corresponding Q&A data through the language model service cluster;

[0009] Forward the Q&A data to the target data source and obtain the user behavior data generated by the target data source based on the Q&A data;

[0010] Perform full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data to generate the call analysis results of the language model service cluster.

[0011] The present invention also provides an information collection and analysis device, including:

[0012] A monitoring module, configured to monitor the language model call requests transmitted by multiple data sources through a gateway with multi-protocol recognition function and a dynamic traffic processing platform; wherein, the data sources at least include a user interface, an application programming interface, and an integrated development environment plug-in;

[0013] A forwarding module, configured to forward the language model call request to the language model service cluster when receiving the language model call request transmitted by the target data source, so as to generate corresponding Q&A data through the language model service cluster;

[0014] An obtaining module, configured to forward the Q&A data to the target data source and obtain the user behavior data generated by the target data source based on the Q&A data;

[0015] An analysis module, configured to perform full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data to generate the call analysis results of the language model service cluster.

[0016] The present invention also provides an electronic device, including: a memory, configured to store a computer program; a processor, configured to implement the steps of any one of the above information collection and analysis methods when executing the computer program.

[0017] The present invention also provides a computer-readable storage medium, in which a computer program is stored, and wherein the computer program implements the steps of any one of the above information collection and analysis methods when being executed by a processor.

[0018] The present invention also provides a computer program product, including a computer program, and the computer program implements the steps of any one of the above information collection and analysis methods when being executed by a processor.

[0019] The information collection and analysis method provided by the present invention monitors the language model call requests transmitted by multiple data sources through a gateway with multi-protocol recognition function and a dynamic traffic processing platform, realizes the unified management of LLM call data, and adapts to multiple protocols; when receiving a language model call request, forwards the language model call request to the language model service cluster to generate corresponding Q&A data, and obtains the user behavior data generated by the target data source based on the Q&A data, so as to perform full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data. The analysis process not only pays attention to the call requests and responses, but also takes into account the corresponding user behavior, making the generated call analysis results more accurate and improving the credibility of the analysis of the LLM application effect.

[0020] In addition, the present invention also provides an information collection and analysis device, equipment, medium and product with the same effect. Brief Description of the Drawings

[0021] In order to more clearly illustrate the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 It is a flowchart of an information collection and analysis method provided by an embodiment of the present invention;

[0023] Figure 2 It is an architecture diagram of an information collection and analysis system provided by an embodiment of the present invention;

[0024] Figure 3 It is a schematic diagram of an information collection and analysis device provided by an embodiment of the present invention. Detailed Embodiments

[0025] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the protection scope of the present invention.

[0026] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0027] In order to enable those skilled in the art of the present technology to better understand the solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] With the rapid development of artificial intelligence technology, the application scenarios of LLM are becoming increasingly rich, covering multiple aspects such as IDE, UI, and API calls. In IDE plugins, developers frequently use LLM for code completion or question answering to improve development efficiency; in UI, users interact with LLM through web pages or applications to obtain an intelligent interaction experience; and through API calls, systems or third-party applications can directly utilize LLM services to achieve a wider range of function integrations.

[0029] However, the current technology faces many challenges in the field of LLM call information collection. Since the call information from different entrances is scattered in various systems and there is a lack of a unified collection mechanism, it is difficult to centrally manage the data, and cross-entrance analysis becomes complex and difficult. At the same time, existing collection technologies often only focus on requests and responses, while ignoring key information such as user behavior, such as the adoption of code completion or UI interaction behavior, which limits the in-depth analysis of the application effect of LLM. These problems together restrict the comprehensive, unified, efficient, and intelligent collection and analysis of LLM call information, and it is difficult to meet the urgent needs of performance evaluation and optimization. Therefore, in order to solve the above problems, the present invention provides an information collection and analysis method. It can be understood that the method provided by the present invention is applied to computing devices such as servers or hosts, which are communicatively connected to multi-entrance data sources (such as UI, API, and IDE plugins) and communicatively connected to an LLM service cluster.

[0030] Figure 1 It is a flowchart of an information collection and analysis method provided by an embodiment of the present invention. As Figure 1 shown, the method includes:

[0031] S10: Monitor the language model call requests transmitted by multiple data sources through a gateway with multi-protocol recognition function and a dynamic traffic processing platform.

[0032] Among them, the data sources at least include user interfaces, application programming interfaces, and integrated development environment plugins.

[0033] Specifically, the present invention pre - constructs a multi - protocol access gateway. Taking the gateway and the dynamic traffic processing platform as the basic software platform, it has the function of multi - protocol recognition. In specific implementation, the gateway and the dynamic traffic processing platform can select high - performance Web server software. For example, open - source software OpenResty or Nginx can be selected as the basic software platform of the multi - protocol access gateway. Utilizing its high performance, high concurrency, and scalability characteristics, it can identify the traffic transmitted through corresponding protocols from different data sources, including but not limited to the traffic of the Hypertext Transfer Protocol version (HTTP), WebSocket protocol, and Server - Sent Events (SEE) protocol, and at the same time identify the language model call requests therein. In this embodiment, there is no limitation on the recognition method of the language model call requests.

[0034] It should be noted that the data sources include at least UI, API, and IDE plugins, and may also include other data sources, which are not limited in this embodiment. In addition, the language model call request is a request to call the LLM service cluster to execute tasks, including but not limited to question - answering tasks and code completion tasks, etc., depending on the specific implementation situation. It should also be noted that the types of language model call requests sent by different data sources may be the same or different.

[0035] S11: When receiving a language model call request transmitted from a target data source, forward the language model call request to the language model service cluster to generate corresponding question - answering data through the language model service cluster.

[0036] When it is confirmed that a language model call request transmitted from a target data source is received, forward the language model call request to the language model service cluster so that the language model service cluster can generate corresponding question - answering data according to the language model call request. For example, when the task in the language model call request is a question, the language model service cluster will generate an answer corresponding to the question; when the task in the language model call request is a code completion task, the language model service cluster will generate a completed code corresponding to the code completion task.

[0037] S12: Forward the question - answering data to the target data source and obtain the user behavior data generated by the target data source based on the question - answering data.

[0038] After the language model service cluster generates the Q&A data corresponding to the language model call request, the Q&A data is forwarded to the target data source, and the user behavior data generated by the target data source based on the Q&A data is obtained. It can be understood that the user behavior data is the behavior of the user according to the Q&A data after receiving the Q&A data. For example, the language model call request of the IDE plugin to the language model service cluster includes a code completion task, and the language model service cluster will generate the corresponding completed code according to the code completion task; when the IDE plugin receives the completed code output by the language model service cluster, the user can choose to adopt or reject the completed code, and adoption / non-adoption is the user behavior corresponding to the completed code. In this embodiment, there is no limitation on the specific manner of obtaining the user behavior data generated by the target data source based on the Q&A data.

[0039] S13: Perform full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data to generate the call analysis result of the language model service cluster.

[0040] Finally, perform full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data. The analysis process not only focuses on the call request and response, but also takes into account the corresponding user behavior, making the generated call analysis result more accurate. In this embodiment, there is no limitation on the specific process of performing full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data, and various analysis methods can be adopted according to the specific implementation situation.

[0041] In this embodiment, through the gateway with multi-protocol recognition function and the dynamic traffic processing platform, the language model call requests transmitted by multiple data sources are monitored, realizing the unified management of LLM call data and adapting to multiple protocols; when a language model call request is received, the language model call request is forwarded to the language model service cluster to generate the corresponding Q&A data, and the user behavior data generated by the target data source based on the Q&A data is obtained, so as to perform full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data. The analysis process not only focuses on the call request and response, but also takes into account the corresponding user behavior, making the generated call analysis result more accurate and improving the credibility of the analysis of the LLM application effect.

[0042] Figure 2 This is the architecture diagram of the information collection and analysis system provided by the embodiment of the present invention. On the basis of the above embodiment, as Figure 2 shown, in order to monitor the language model call request, in some embodiments, through the gateway with multi-protocol recognition function and the dynamic traffic processing platform, the language model call requests transmitted by multiple data sources are monitored, including:

[0043] S101: Monitor the traffic data transmitted by each data source through the corresponding protocol based on the gateway and the dynamic traffic processing platform.

[0044] S102: Identify the language model call requests that conform to the language model call characteristics in each traffic data through deep packet inspection technology.

[0045] In order to be able to identify language model call requests, in this embodiment, the gateway and the dynamic traffic processing platform monitor the traffic data transmitted by each data source through the corresponding protocol, and identify the language model call requests that conform to the language model call characteristics in each traffic data through deep packet inspection (DPI) technology.

[0046] It should be noted that the process of the DPI technology identifying language model call requests in traffic data involves in-depth analysis of data packets. When a data packet passes through a network device, the DPI system extracts and checks the header and payload information of the data packet. It uses predefined rules, feature libraries or machine learning algorithms to identify the traffic patterns of specific applications or services. For example, by analyzing the URL, User-Agent string or characteristics of a specific protocol in an HTTP request, DPI can determine whether the request is from a web browser, an email client or other applications. In addition, DPI can also identify specific patterns or metadata in encrypted traffic to infer its content or purpose. In this way, the DPI technology can accurately identify and classify language model call requests in various network traffics.

[0047] Specifically, the gateway and the dynamic traffic processing platform use DPI technology to intelligently identify requests that conform to the model call characteristics in various protocols, such as multi-dimensionally and intelligently identifying requests that conform to the model call characteristics through URL path characteristics, request header characteristics, request body characteristics, etc., improving the identification accuracy and reducing misjudgment.

[0048] Based on the above embodiments, in order to accurately forward the language model call requests to the LLM service cluster, in some embodiments, when receiving the language model call requests transmitted by the target data source, forwarding the language model call requests to the language model service cluster includes:

[0049] S103: Generate a request fingerprint of the language model call request through a preset identifier generation algorithm.

[0050] Among them, the request fingerprint at least includes metadata information such as the user Internet protocol address, client type, entry type, model version and request timestamp.

[0051] S104: Add the request fingerprint to the request header of the language model call request.

[0052] S105: Forward the language model call request to the language model service cluster according to the request fingerprint in the corresponding request header.

[0053] Specifically, a request fingerprint generation module is developed in the multi-protocol access gateway in advance based on the Lua script (a lightweight, efficient, and embeddable scripting language). This module can generate the request fingerprint of the language model call request through a preset identifier generation algorithm during operation. For example, the Universally Unique Identifier Generation Algorithm (UUID) is used to generate the request fingerprint of the language model call request. It should be noted that UUID is a 128-bit number, usually represented by 32 hexadecimal characters, with extremely high uniqueness, almost guaranteeing no duplication. When generating the request fingerprint, a UUID can be created at the time of request initiation and sent as a parameter or header information of the request. For example, in an HTTP request, the UUID can be attached to the request as a custom header (such as X-Request-ID). This ensures that each request will have a unique identifier, facilitating internal system tracking, debugging, and logging. In addition, the request fingerprint includes at least the metadata information of the user's Internet protocol address, client type, entry type, model version, and request timestamp, and may also include other metadata information, which is not limited in this embodiment.

[0054] Subsequently, add the request fingerprint to the request header of the language model call request, thus generating a language model call request containing the request fingerprint. Finally, forward the language model call request to the LLM service cluster according to the request fingerprint in the corresponding request header. The request fingerprint, as the key identifier for full-link data association, can solve the problems of complex heterogeneous protocol adaptation and missing full-link tracing, ensuring the complete transmission of data.

[0055] In some embodiments, to forward the language model call request to the language model service cluster, forwarding the language model call request to the language model service cluster according to the request fingerprint in the corresponding request header includes:

[0056] S106: Configure the proxy transfer instruction between the gateway and the dynamic traffic processing platform.

[0057] S107: Based on the proxy transfer instruction, control the gateway and the dynamic traffic processing platform to forward the language model call request to the language model service cluster according to the request fingerprint, entry type, and model version.

[0058] Specifically, configure the proxy_pass instruction of the gateway and the dynamic traffic processing platform. Based on the proxy_pass instruction, control the gateway and the dynamic traffic processing platform to flexibly route and forward the language model call requests to the backend LLM service cluster according to the request fingerprint, entry type, and model version, achieving load balancing.

[0059] In summary, the multi-protocol access gateway uses the gateway and the dynamic traffic processing platform as the basic software platform, which is the unified traffic entry of the system, responsible for functions such as protocol adaptive recognition, multi-protocol access, request interception and forwarding, and dynamic request fingerprint generation. It solves the problems of complex heterogeneous protocol adaptation and lack of full-link tracing, and ensures the integrity and real-time nature of streaming data. In addition, to ensure the normal operation of the gateway and the dynamic traffic processing platform, the configured gateway and dynamic traffic processing platform can also be deployed to an independent server or containerized environment for performance testing and functional verification to ensure the high performance, high reliability, and high availability of the gateway.

[0060] As can be seen from the above embodiments, the data sources at least include UI, API, and IDE plugins. Whether each data source will trigger an action after receiving the Q&A data from the LLM service cluster mainly depends on the design purpose and system permissions. Specifically, UI and API generally only display and output the Q&A data without generating user actions, while the IDE plugin may generate corresponding user actions according to the Q&A data (such as code data), such as adopting or not adopting the code. Therefore, in some embodiments, forwarding the Q&A data to the target data source and obtaining the user action data generated by the target data source based on the Q&A data includes:

[0061] S120: When the target data source is the user interface or the application programming interface, forward the streaming Q&A data generated by the language model service cluster to the corresponding user interface or application programming interface, and aggregate the streaming Q&A data into the corresponding complete Q&A data.

[0062] S121: When the target data source is the integrated development environment plugin, forward the streaming Q&A data generated by the language model service cluster to the corresponding integrated development environment plugin, and aggregate the streaming Q&A data into the corresponding complete Q&A data.

[0063] S122: Obtain the user action data generated by the integrated development environment plugin based on the corresponding streaming Q&A data.

[0064] Specifically, when the target data source is a UI or API, the streaming Q&A data generated by the language model service cluster is forwarded to the corresponding UI or API, and the streaming Q&A data is aggregated into the corresponding complete Q&A data. When the target data source is an IDE plugin, the streaming Q&A data generated by the LLM service cluster is forwarded to the corresponding IDE plugin, and the streaming Q&A data is aggregated into the corresponding complete Q&A data. At the same time, the user behavior data generated by the IDE plugin based on the corresponding streaming Q&A data is obtained.

[0065] It should be noted that the LLM service cluster outputs Q&A data using streaming data (i.e., generating and transmitting word by word) mainly for efficiency and experience optimization. Since it takes time to generate a complete answer, streaming transmission can gradually return the results, reducing the "blank period" for users to wait (such as the chat conversation showing word by word), enhancing the sense of real-time interaction. At the same time, this method saves memory resources, avoids the overhead of processing long texts at once, and supports interruption in the middle (such as when the user stops the request). Technically, the model generates content through word-by-word probability prediction, the server can generate and push while the client synchronously renders, which is especially suitable for web applications and real-time scenarios (such as translation, code completion). Therefore, in order to enable the target data source to obtain the generated answer in a timely manner and improve the answer transmission efficiency, it is necessary to directly forward the streaming Q&A data to the target data source. At the same time, in order to better analyze the Q&A data output by the LLM service cluster, it is necessary to aggregate the streaming Q&A data into the corresponding complete Q&A data, so as to analyze the complete Q&A data. In this embodiment, the specific process of aggregating the streaming Q&A data into the corresponding complete Q&A data is not limited.

[0066] Based on the above embodiments, in some embodiments, aggregating the streaming Q&A data into the corresponding complete Q&A data includes:

[0067] S123: Build a streaming data processing platform and configure the data protocol, asynchronous processing, and concurrency parameters of the streaming data processing platform;

[0068] S124: Receive and parse the streaming Q&A data generated by the language model service cluster through the streaming data processing platform;

[0069] S125: Extract each target business field in the streaming Q&A data through regular expressions;

[0070] S126: Real-time splice each target business field transmitted in chunks through a content splicing algorithm and add metadata information to each target business field to generate the corresponding complete Q&A data;

[0071] Among them, the metadata information includes at least a timestamp, a block sequence number, and a data source.

[0072] Such as Figure 2As shown in the figure, in order to aggregate the streaming Q&A data into the corresponding complete Q&A data, a streaming data processing platform is pre-built in this embodiment to process the streaming data. It should be noted that the streaming data processing platform can be built by selecting a high-performance streaming processing framework. For example, open-source streaming processing frameworks such as Apache Kafka Streams, Apache Flink, or RedisStream, or lightweight streaming processing libraries can be selected. At the same time, configure the data protocol of the streaming data processing platform. Specifically, based on the API provided by the streaming processing framework or library, implement the reception and parsing of the SSE protocol for each block of data stream. It is also necessary to configure the asynchronous processing and concurrency parameters of the streaming processing framework or library, such as the thread pool size and concurrency degree, to optimize the data processing performance.

[0073] Furthermore, by receiving and parsing the streaming Q&A data generated by the LLM service cluster through the streaming data processing platform, and extracting each target business field in the streaming Q&A data through regular expressions, such as response content, timestamp, user Internet protocol address, session identifier, etc., the data parsing efficiency and accuracy can be improved.

[0074] Finally, the content splicing algorithm is used to splice each target business field transmitted in blocks in real time. In this embodiment, there is no limitation on the selected content splicing algorithm. For example, StringBuilder or StringBuffer can be selected to splice the response content transmitted in blocks in real time. During the streaming processing, metadata information is added to each target business field to generate the corresponding complete Q&A data. It should be noted that the metadata information includes at least a timestamp, a block number, and a data source.

[0075] In addition, in order to ensure the reliability of the streaming data processing platform, after the initial construction is completed, the streaming data processing engine can also be deployed to a cluster environment for performance testing and stress testing to ensure the high throughput, low latency, and high reliability of the engine.

[0076] In this embodiment, the pre-built streaming data processing platform realizes the streaming data processing. By adopting a series of high-performance real-time parsing and asynchronous aggregation technologies such as SSE protocol block-by-block parsing, regular expression feature extraction, response content dynamic splicing, block metadata marking, asynchronous processing, and concurrency mechanism, the low-latency processing of a large amount of real-time data streams is realized.

[0077] In order to monitor the code completion events of users and collect the user behavior data generated by users based on the streaming Q&A data, on the basis of the above embodiments, in some embodiments, the user behavior data generated by the integrated development environment plugin according to the corresponding streaming Q&A data is obtained, including:

[0078] S130: Monitor the code completion events of users through the integrated development environment plugin.

[0079] S131: When receiving the streaming Q&A data generated by the language model service cluster through the integrated development environment plugin, determine that a code completion event is triggered, and determine code completion suggestions based on the streaming Q&A data.

[0080] S132: Determine whether the user accepts the code completion suggestions within a preset period; if so, proceed to step S133; if not, proceed to step S134.

[0081] S133: Generate user behavior data representing that the user adopts the code completion suggestions.

[0082] S134: Generate user behavior data representing that the user rejects the code completion suggestions.

[0083] As Figure 2 shown, in this embodiment, the event listening interface provided by the IDE plugin development framework is used to listen for code completion events in the code editor through the behavior correlation collector, such as the user entering keywords, triggering the display of code completion suggestions, etc. When receiving the streaming Q&A data generated by the LLM service cluster through the IDE plugin, determine that a code completion event is triggered, and determine code completion suggestions based on the streaming Q&A data.

[0084] Further, the behavior collection module of the behavior correlation collector determines whether the user accepts the code completion suggestions within a preset period. If it is confirmed that the user accepts the code completion suggestions within the preset period, generate user behavior data representing that the user adopts the code completion suggestions. If it is confirmed that the user does not accept the code completion suggestions within the preset period, generate user behavior data representing that the user rejects the code completion suggestions.

[0085] It should be noted that in this embodiment, there is no limit to the preset period. For example, it can be set to 10s, that is, when the completion suggestions are displayed, start a 10s timer. If the user does not perform an adoption operation (such as not selecting the suggestions or not performing code editing) within 10s, it is determined that the user rejects the adoption.

[0086] In summary, in this embodiment, through the IDE plugin integration and delayed recording technology, the accurate collection and delayed determination of IDE code completion user behavior (explicit adoption and implicit rejection) are realized.

[0087] In order to better store the Q&A data of the IDE plugin and associate the Q&A data with the corresponding user behavior, based on the above embodiment, in some embodiments, after obtaining the user behavior data generated by the integrated development environment plugin according to the corresponding streaming Q&A data, it further includes:

[0088] S135: Set up a cache space and configure the survival time of cache entries in the cache space.

[0089] S136: Store the fingerprint mapping table based on the cache space.

[0090] Among them, the fingerprint mapping table contains multiple key-value pairs. The key represents the request fingerprint, and the value represents the complete Q&A data corresponding to the request fingerprint.

[0091] S137: Determine the request fingerprint of the user behavior data.

[0092] S138: Determine the key-value pair corresponding to the user behavior data in the fingerprint mapping table according to the request fingerprint of the user behavior data.

[0093] S139: Generate behavior Q&A association data according to the user behavior data and the corresponding key-value pair.

[0094] Specifically, build the behavior association module of the behavior association collector, select a high-performance in-memory database, such as Redis or Memcached, to build the cache space, and at the same time configure the least recently used (LRU) policy of the cache space.

[0095] It should be noted that LRU is a common cache eviction algorithm used to manage data in the cache to optimize performance and resource utilization. The core idea of the LRU algorithm is: when the cache capacity reaches the upper limit, the data item that has been least recently accessed is preferentially removed. In this way, the data usually retained in the cache is the data that has been frequently accessed recently, thereby improving the cache hit rate and system performance. In the LRU algorithm, each data item will record the time or order of its last access. When data needs to be evicted, the algorithm will select the data item that has not been accessed for the longest time for removal. The implementation of the LRU algorithm usually uses a combination of a doubly linked list and a hash table to achieve efficient access and update operations. This algorithm is widely used in various cache systems, such as virtual memory management in operating systems, database query caches, and page caches in web applications. By using the LRU algorithm, the system can more effectively utilize limited cache resources and improve overall performance and response speed.

[0096] Specifically, configure the survival time of the cache items in the cache space based on the LRU policy, for example, set the survival time to 30s. Dynamically evict cold data through the LRU algorithm, combined with an accurate 30s survival period, to maintain a high cache hit rate while ensuring data timeliness and preventing memory overflow.

[0097] Furthermore, store the fingerprint mapping table based on the cache space. It is worth noting that the fingerprint mapping table is a mapping relationship table between the request fingerprint and the model call data, used to cache the request context information in the recent period of time, and specifically contains multiple key-value pairs (key-value). The key represents the request fingerprint, and the value represents the complete Q&A data corresponding to the request fingerprint. In specific implementation, a hash table can be used to construct the fingerprint mapping table.

[0098] Subsequently, the request fingerprint of the user behavior data is determined to facilitate determining the key-value pair corresponding to the user behavior data in the fingerprint mapping table according to the request fingerprint of the user behavior data. In this embodiment, there is no limitation on the determination method of the request fingerprint of the user behavior data, which depends on the specific implementation. Finally, after determining the key-value pair corresponding to the user behavior data, behavior-question-and-answer association data is generated according to the user behavior data and the corresponding key-value pair, so as to associate the question-and-answer data with the corresponding user behavior for subsequent analysis.

[0099] In this embodiment, by building a behavior association collector and proposing a request fingerprint-behavior association algorithm, the association between user behavior data and model call data is established, and a full-link user behavior model is constructed to facilitate the subsequent complete analysis of the model call situation.

[0100] Based on the above embodiments, in some embodiments, determining the request fingerprint of the user behavior data includes:

[0101] S140: When receiving the streaming question-and-answer data generated by the language model service cluster, determine the corresponding question-and-answer timestamp through the integrated development environment plug-in.

[0102] S141: Generate the request fingerprint of the user behavior data according to the generated user behavior data, the corresponding question-and-answer timestamp, and the preset identifier generation algorithm.

[0103] To determine the request fingerprint of the user behavior data, specifically in this embodiment, when receiving the streaming question-and-answer data generated by the LLM service cluster and completing a round of conversation, the IDE plug-in will generate and determine the corresponding question-and-answer timestamp. Subsequently, the IDE plug-in will generate the request fingerprint of the user behavior data according to the generated user behavior data, the corresponding question-and-answer timestamp, and the preset identifier generation algorithm.

[0104] It should be noted that the preset identifier generation algorithm here should be the same as the preset identifier generation algorithm for generating the request fingerprint of the model call request, so as to ensure the generation of the same request fingerprint to facilitate the association of the question-and-answer data with the corresponding user behavior.

[0105] Before performing the full-link multi-dimensional analysis based on the question-and-answer data and the corresponding user behavior data, after obtaining the user behavior data generated by the target data source according to the question-and-answer data, it further includes:

[0106] S142: When the target data source is the user interface or the application programming interface, store the corresponding complete question-and-answer data in the pre-constructed storage space.

[0107] S143: When the target data source is an integrated development environment plugin, store the corresponding behavior Q&A association data in a pre-constructed storage space.

[0108] To better store the complete Q&A data / behavior Q&A association data, in this embodiment, when the target data source is UI or API, store the corresponding complete Q&A data in a pre-constructed storage space. When the target data source is an IDE plugin, store the corresponding behavior Q&A association data in a pre-constructed storage space. The storage process is described in detail below:

[0109] In some embodiments, storing the complete Q&A data / behavior Q&A association data in the storage space includes:

[0110] S144: Set up a hot data storage space, a warm data storage space, and a cold data storage space.

[0111] S145: Store the complete Q&A data / behavior Q&A association data in the hot data storage space.

[0112] S146: Monitor the access information of the complete Q&A data / behavior Q&A association data.

[0113] S147: Degrade and store the complete Q&A data / behavior Q&A association data in the warm data storage space or the cold data storage space according to the access information.

[0114] As Figure 2 shown, in a specific implementation, deploy an intelligent storage controller to set up a hot data storage space, a warm data storage space, and a cold data storage space respectively. In this embodiment, there is no limitation on the storage media selected for each storage space. For example, Redis Cluster or Memcached Cluster can be selected as the hot data storage space, Apache Kafka or RabbitMQ can be selected as the warm data storage space, and ClickHouse Cluster or HDFS Cluster can be selected as the cold data storage space.

[0115] Further configure the hierarchical storage policy. Specifically, store the complete Q&A data / behavior Q&A association data in the hot data storage space, and then monitor the access information of the complete Q&A data / behavior Q&A association data. It should be noted that the access information at least includes the access frequency. Finally, degrade and store the complete Q&A data / behavior Q&A association data in the warm data storage space or the cold data storage space according to the access information. It should be noted that in this embodiment, there is no limitation on the specific process of degrading and storing the complete Q&A data / behavior Q&A association data in the warm data storage space or the cold data storage space, which depends on the specific implementation situation.

[0116] In addition, to ensure the availability of the intelligent storage controller, the intelligent storage controller can also be deployed in a cluster environment for performance testing, stress testing, and disaster recovery testing to ensure the high performance, high reliability, high scalability, and data security of the storage controller.

[0117] In this embodiment, by deploying the intelligent storage controller, constructing a data hierarchical storage system using hierarchical storage media, and creating a dynamic degradation mechanism, the efficient storage management of data is ensured.

[0118] Based on the above embodiment, in some embodiments, downgrading the complete Q&A data / behavior Q&A associated data to the warm data storage space or the cold data storage space according to the access information includes:

[0119] S148: Determine whether there is an access to the complete Q&A data / behavior Q&A associated data within a first preset time according to the access information; if so, return to step S148; if not, proceed to step S149.

[0120] S149: Downgrade the complete Q&A data / behavior Q&A associated data to the warm data storage space, and determine whether there is an access to the complete Q&A data / behavior Q&A associated data within a second preset time according to the access information; if so, return to step S149; if not, proceed to step S150.

[0121] S150: Downgrade the complete Q&A data / behavior Q&A associated data to the cold data storage space.

[0122] Wherein, the first preset time is less than the second preset time.

[0123] To implement the downgraded storage of the complete Q&A data / behavior Q&A associated data, in this embodiment, it is specifically determined according to the access information whether there is an access to the complete Q&A data / behavior Q&A associated data within a first preset time.

[0124] If it is confirmed that there is an access to the complete Q&A data / behavior Q&A associated data within the first preset time, then the complete Q&A data / behavior Q&A associated data is considered as high-frequency access data and still needs to be stored in the hot data storage space, and return to the step of determining whether there is an access to the complete Q&A data / behavior Q&A associated data within the first preset time according to the access information to continue monitoring the access information of the data. If it is confirmed that there is no access to the complete Q&A data / behavior Q&A associated data within the first preset time, then the access frequency of the complete Q&A data / behavior Q&A associated data is considered average. To optimize data storage, it is necessary to downgrade the complete Q&A data / behavior Q&A associated data to the warm data storage space and determine whether there is an access to the complete Q&A data / behavior Q&A associated data within a second preset time according to the access information.

[0125] If it is confirmed that there is an access to the complete Q&A data / behavior Q&A association data within the second preset time, it is considered that the access frequency of the complete Q&A data / behavior Q&A association data is average, and it still needs to be stored in the warm data storage space. Return to the step of judging whether there is an access to the complete Q&A data / behavior Q&A association data within the second preset time according to the access information to continue monitoring the access information of the data. If it is confirmed that there is no access to the complete Q&A data / behavior Q&A association data within the second preset time, it is considered that the access frequency of the complete Q&A data / behavior Q&A association data is low. In order to optimize data storage, it is necessary to downgrade and store the complete Q&A data / behavior Q&A association data to the cold data storage space. In this way, the downgraded storage of data is achieved.

[0126] It should be noted that in this embodiment, there are no restrictions on the first preset time and the second preset time, as long as it is ensured that the first preset time is less than the second preset time.

[0127] Based on the above embodiments, in some embodiments, a full-link multi-dimensional analysis is performed on the Q&A data and the corresponding user behavior data, including:

[0128] S151: Build a data analysis platform and the corresponding data warehouse.

[0129] S152: Obtain the complete Q&A data / behavior Q&A association data in the storage space and store it in the data warehouse.

[0130] S153: Perform multi-dimensional analysis on the complete Q&A data / behavior Q&A association data through the data analysis platform to generate the call analysis results of the language model service cluster.

[0131] As Figure 2 shown, in order to implement the data analysis of LLM calls, a data analysis platform and the corresponding data warehouse are specifically built. For example, ClickHouse Cluster or Hadoop+Spark / Flink is selected as the data warehouse of the data analysis platform, and other frameworks can also be used, which are not restricted in this embodiment.

[0132] Subsequently, the data analysis platform obtains the complete Q&A data / behavior Q&A association data in the storage space and stores it in the data warehouse. Finally, multi-dimensional analysis is performed on the complete Q&A data / behavior Q&A association data to generate the call analysis results of the language model service cluster. The data storage and analysis process will be described in detail below:

[0133] In some embodiments, obtaining the complete Q&A data / behavior Q&A association data in the storage space and storing it in the data warehouse includes:

[0134] S154: Clean and standardize the complete Q&A data / behavior Q&A association data through the data analysis platform to obtain the preprocessed complete Q&A data / behavior Q&A association data.

[0135] S155: Store the preprocessed complete Q&A data / behavior Q&A association data in the data warehouse.

[0136] Specifically, based on big data processing frameworks such as Spark and Flink in the data analysis platform, implement data cleaning, transformation, and standardization processing for the complete Q&A data / behavior Q&A association data. For example, data format conversion, outlier processing, data desensitization, data normalization, and efficiently store the cleaned structured data in the data warehouse.

[0137] Correspondingly, conduct multi-dimensional analysis on the complete Q&A data / behavior Q&A association data through the data analysis platform, including:

[0138] S156: Conduct multi-dimensional analysis on the complete Q&A data / behavior Q&A association data through various data processing and analysis algorithms to generate call analysis results.

[0139] The data analysis platform uses various data processing and analysis algorithms to conduct multi-dimensional analysis on the complete Q&A data / behavior Q&A association data. The specific analysis methods include but are not limited to call volume statistical analysis, user behavior analysis, model performance evaluation, real-time monitoring and early warning, A / B testing and effect evaluation, error and exception analysis, resource consumption analysis, security and compliance auditing, etc., and finally generate the call analysis results of the model. The analysis process is described below:

[0140] First, for the LLM service cluster, call volume statistical analysis can reveal the frequency at which the model is requested, peak usage periods, and the popularity of different functional modules. This data helps optimize model deployment, ensure stable service during high demand, and guide future model expansion and resource planning. Second, by analyzing user interactions with the LLM, including query types, feedback, and usage habits, a deep understanding of user needs and preferences can be gained, which can be used to improve the model's response quality, optimize the user interface, and develop new functions that better meet user expectations. On the other hand, for the performance evaluation of the model in terms of accuracy, response time, relevance, and fluency of the generated text; by regularly evaluating and comparing different versions or configurations of the model, areas for improvement can be identified to ensure that the model continuously provides high-quality output. Subsequently, real-time monitoring of the LLM's operating status, including resource usage, response latency, and error rate, helps detect and resolve potential problems in a timely manner; meanwhile, by setting up a warning mechanism, a quick response can be made when performance degrades or a failure occurs, ensuring the continuity and reliability of the service. In addition, through A / B testing, the impact of different versions of the LLM or specific functions on the user experience and business metrics can be compared, which helps determine the best model configuration or function design, optimize user satisfaction, and increase the value of the model in practical applications. Further, collecting and analyzing errors and anomalies generated during the operation of the LLM can identify common problem patterns and potential defects; by fixing these problems, the stability and robustness of the model can be improved, and the errors and interruptions encountered by users can be reduced. Monitoring the resource consumption of the LLM, such as computing resources, storage, and bandwidth usage, helps optimize resource allocation and cost management; by analyzing resource usage patterns, optimization opportunities can be identified, resource utilization can be increased, and the model can be ensured to operate efficiently within the budget. Finally, conducting security and compliance audits on the LLM to ensure that it complies with relevant regulations and standards when processing user data and generating content. This includes reviewing data privacy protection measures, content filtering mechanisms, and access control policies to protect user rights and avoid legal risks. Through continuous audits and improvements, user trust can be established, and the model can be ensured to operate in a secure and compliant environment.

[0141] In this embodiment, the data analysis platform performs data reception, cleaning and storage, call volume statistical analysis, user behavior analysis, model performance evaluation, real-time monitoring and warning, A / B testing and effect evaluation, error and anomaly analysis, resource consumption analysis, and security and compliance audits on the complete Q&A data / behavior Q&A association data, providing multi-dimensional, intelligent, and comprehensive data analysis capabilities.

[0142] In addition, to ensure the reliability of the data analysis platform, it can also be deployed in a cluster environment for functional testing, performance testing, stress testing, and stability testing to ensure that the platform's data processing and analysis capabilities, multi-dimensional intelligent analysis functions, and real-time monitoring and warning capabilities have high concurrency, high reliability, and high availability.

[0143] To enable users to view the call analysis results of the language model more intuitively, based on the above embodiments, in some embodiments, after generating the call analysis results of the language model service cluster, it further includes:

[0144] S157: Develop a data visualization display interface for the data analysis platform through a data visualization tool.

[0145] S158: Output the call analysis results through the data visualization display interface.

[0146] Specifically, based on the data analysis platform, select open-source data visualization tools to develop data visualization display interfaces such as diversified and customizable visual reports, dashboards, and monitoring dashboards to present the data analysis results in real time. For example, call volume trend charts, user behavior distribution charts, model performance indicator dashboards, real-time monitoring dashboards, A / B test comparison reports, etc. Support customizable visual reports and dashboards, and users can flexibly configure and display key indicators according to their needs. Lower the threshold for data understanding and use, facilitate users to intuitively and clearly understand the application situation, performance performance, and potential problems of the LLM, and provide data support for decision-making.

[0147] In addition, to facilitate users to query, analyze, and use the collected LLM call data and analysis results, the method further includes:

[0148] S159: Develop a data analysis interface for the data analysis platform based on the web application programming interface framework and open the data analysis function of the data analysis platform.

[0149] S160: Develop a visualization interface based on the front-end framework so that users can directly access the call analysis results in the data analysis platform through the visualization interface and the data analysis interface.

[0150] Specifically, based on open-source web application programming interface frameworks, such as API frameworks like FastAPI, Spring Boot, Django REST framework, etc., develop data analysis interfaces (RESTful APIs) for the data analysis platform and open up the data analysis functions of the data analysis platform. Provide multi-dimensional data filtering, sorting, and pagination functions, such as call volume query API, user behavior analysis API, model performance evaluation API, real-time monitoring data API, A / B test result API, etc. Facilitate users to obtain the required data on demand and support SDKs for multiple programming languages and development platforms, reducing the threshold for using the API. At the same time, support JSON data format and multiple programming languages.

[0151] Furthermore, develop a visual interface based on a front-end framework to enable users to directly access the call analysis results in the data analysis platform through the visual interface and data analysis interfaces. For example, select front-end frameworks such as React, Vue, Angular, etc., to develop a web-based visual interface. Users can directly access the data analysis platform through a browser, interactively query, analyze, and visualize LLM call data, customize visual reports and dashboards, and export data analysis reports.

[0152] In addition, to ensure the availability of the data analysis interfaces, the data analysis interfaces can be deployed to a web server cluster for functional testing, performance testing, and security testing to ensure the ease of use, high performance, high reliability, and high security of the interfaces.

[0153] In this embodiment, by externally providing standardized API interfaces and visual interfaces, it is convenient for users to query, analyze, and use the collected LLM call data and analysis results, realizing the open sharing and application of data.

[0154] To enable the LLM service cluster to operate better, in some embodiments, after generating the call analysis results of the language model service cluster, it further includes:

[0155] S161: Obtain the load information of the language model service cluster;

[0156] S162: Generate an allocation strategy for language model call requests based on the call analysis results and load information, so as to process new language model call requests based on the allocation strategy.

[0157] Specifically, obtaining the load information of the language model service cluster includes, but is not limited to, processor usage rate, memory usage, disk input / output volume, network traffic, etc. Subsequently, dynamically adjust the request allocation strategy according to the call analysis results and load information, so as to process new language model call requests based on the allocation strategy.

[0158] In addition, in some embodiments, it further includes:

[0159] S163: Monitor the health status of the language model service cluster.

[0160] S164: When a faulty server is detected in the language model service cluster, transfer the language model call requests being processed in the faulty server to the remaining healthy servers for execution.

[0161] Specifically, regularly check the health status of the LLM service cluster, promptly detect and handle faulty servers to ensure the high availability of the system. When a server in the LLM service cluster fails, transfer the language model call requests being processed in the faulty server to the remaining healthy servers for execution, thus avoiding interruption of request processing.

[0162] In summary, performing load balancing and fault handling on the LLM service cluster based on the call analysis results can effectively improve the performance, stability, and resource utilization rate of the LLM, thereby better meeting user needs and business requirements.

[0163] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0164] Figure 3 It is a schematic diagram of an information collection and analysis device provided by an embodiment of the present invention. As Figure 3 shown, the device includes:

[0165] A monitoring module 10, configured to monitor the language model call requests transmitted by multiple data sources through a gateway with multi - protocol recognition function and a dynamic traffic processing platform; wherein, the data sources at least include a user interface, an application programming interface, and an integrated development environment plug - in.

[0166] A forwarding module 11, configured to forward the language model call request to the language model service cluster when receiving the language model call request transmitted by the target data source, so as to generate corresponding Q&A data through the language model service cluster.

[0167] An acquisition module 12, configured to forward the Q&A data to the target data source and acquire the user behavior data generated by the target data source based on the Q&A data.

[0168] An analysis module 13, configured to perform full - link multi - dimensional analysis based on the Q&A data and the corresponding user behavior data to generate a call analysis result of the language model service cluster.

[0169] In some embodiments, the monitoring module 10 includes:

[0170] The first monitoring sub-module is used to monitor the traffic data transmitted by each data source through the corresponding protocol based on the gateway and the dynamic traffic processing platform;

[0171] The first identification sub-module is used to identify the language model call requests that conform to the language model call characteristics in each traffic data through the deep packet detection technology.

[0172] In some embodiments, the forwarding module 11 includes:

[0173] The first generation sub-module is used to generate a request fingerprint for the language model call request through a preset identifier generation algorithm; wherein, the request fingerprint at least includes metadata information such as the user Internet protocol address, client type, entry type, model version, and request timestamp;

[0174] The first addition sub-module is used to add the request fingerprint to the request header of the language model call request;

[0175] The first forwarding sub-module is used to forward the language model call request to the language model service cluster according to the request fingerprint in the corresponding request header.

[0176] In some embodiments, the first forwarding sub-module includes:

[0177] The first configuration sub-module is used to configure the proxy transfer instruction of the gateway and the dynamic traffic processing platform;

[0178] The second forwarding sub-module is used to control the gateway and the dynamic traffic processing platform to forward the language model call request to the language model service cluster according to the request fingerprint, entry type, and model version based on the proxy transfer instruction.

[0179] In some embodiments, the acquisition module 12 includes:

[0180] The first aggregation sub-module is used to forward the streaming Q&A data generated by the language model service cluster to the corresponding user interface or application programming interface when the target data source is the user interface or application programming interface, and aggregate the streaming Q&A data into the corresponding complete Q&A data;

[0181] The second aggregation sub-module is used to forward the streaming Q&A data generated by the language model service cluster to the corresponding integrated development environment plug-in when the target data source is the integrated development environment plug-in, and aggregate the streaming Q&A data into the corresponding complete Q&A data;

[0182] The first acquisition sub-module is used to acquire the user behavior data generated by the integrated development environment plug-in based on the corresponding streaming Q&A data.

[0183] In some embodiments, the first aggregation sub-module and the second aggregation sub-module include:

[0184] The first building sub-module is used to build a streaming data processing platform and configure the data protocol, asynchronous processing, and concurrency parameters of the streaming data processing platform;

[0185] The first parsing sub-module is used to receive and parse the streaming Q&A data generated by the language model service cluster through the streaming data processing platform;

[0186] The first extraction sub-module is used to extract each target business field in the streaming Q&A data through regular expressions;

[0187] The splicing sub-module is used to splice each target business field transmitted in chunks in real time through a content splicing algorithm and add metadata information to each target business field to generate corresponding complete Q&A data;

[0188] Among them, the metadata information at least includes a timestamp, a block serial number, and a data source.

[0189] In some embodiments, the first acquisition sub-module includes:

[0190] The first monitoring sub-module is used to monitor the code completion event of the user through the integrated development environment plug-in;

[0191] The first determination sub-module is used to determine that the code completion event is triggered when receiving the streaming Q&A data generated by the language model service cluster through the integrated development environment plug-in, and determine the code completion suggestion according to the streaming Q&A data;

[0192] The first judgment sub-module is used to judge whether the user accepts the code completion suggestion within a preset period; if so, generate user behavior data representing that the user adopts the code completion suggestion; if not, generate user behavior data representing that the user rejects the code completion suggestion.

[0193] In some embodiments, it further includes:

[0194] The second building sub-module is used to build a cache space and configure the survival time of cache items in the cache space;

[0195] The first storage sub-module is used to store the fingerprint mapping table based on the cache space; among them, the fingerprint mapping table contains multiple key-value pairs, the key represents the request fingerprint, and the value represents the complete Q&A data corresponding to the request fingerprint;

[0196] The second determination sub-module is used to determine the request fingerprint of the user behavior data;

[0197] The third determination sub-module is used to determine the key-value pair corresponding to the user behavior data in the fingerprint mapping table according to the request fingerprint of the user behavior data;

[0198] Generate behavior Q&A association data based on user behavior data and corresponding key-value pairs.

[0199] In some embodiments, the second determination sub-module includes:

[0200] The fourth determination sub-module is configured to determine the corresponding Q&A timestamp through an integrated development environment plug-in when receiving the streaming Q&A data generated by the language model service cluster;

[0201] The second generation sub-module is configured to generate a request fingerprint of the user behavior data according to the generated user behavior data, the corresponding Q&A timestamp, and a preset identifier generation algorithm.

[0202] In some embodiments, it further includes:

[0203] The second storage sub-module is configured to store the corresponding complete Q&A data in a pre-constructed storage space when the target data source is a user interface or an application programming interface;

[0204] The third storage sub-module is configured to store the corresponding behavior Q&A association data in a pre-constructed storage space when the target data source is an integrated development environment plug-in.

[0205] In some embodiments, the second storage sub-module and the third storage sub-module include:

[0206] The third construction sub-module is configured to construct a hot data storage space, a warm data storage space, and a cold data storage space;

[0207] The fourth storage sub-module is configured to store the complete Q&A data / behavior Q&A association data in the hot data storage space;

[0208] The second monitoring sub-module is configured to monitor the access information of the complete Q&A data / behavior Q&A association data;

[0209] The downgraded storage sub-module is configured to downgrade and store the complete Q&A data / behavior Q&A association data in the warm data storage space or the cold data storage space according to the access information.

[0210] In some embodiments, the downgraded storage sub-module includes:

[0211] The second judgment sub-module is configured to judge whether there is an access to the complete Q&A data / behavior Q&A association data within a first preset time according to the access information; if so, trigger the second judgment sub-module; if not, trigger the third judgment sub-module;

[0212] The third judgment sub-module is used to demote and store the complete Q&A data / behavior Q&A association data to the warm data storage space, and judge whether there is an access to the complete Q&A data / behavior Q&A association data within the second preset time according to the access information; if so, trigger the third judgment sub-module; if not, demote and store the complete Q&A data / behavior Q&A association data to the cold data storage space.

[0213] Wherein, the first preset time is less than the second preset time.

[0214] In some embodiments, the analysis module 13 includes:

[0215] The fourth construction sub-module is used to construct a data analysis platform and a corresponding data warehouse;

[0216] The second acquisition sub-module is used to acquire the complete Q&A data / behavior Q&A association data in the storage space and store it in the data warehouse;

[0217] The first analysis sub-module is used to perform multi-dimensional analysis on the complete Q&A data / behavior Q&A association data through the data analysis platform to generate a call analysis result of the language model service cluster.

[0218] In some embodiments, the second acquisition sub-module includes:

[0219] The data preprocessing sub-module is used to perform data cleaning and standardization processing on the complete Q&A data / behavior Q&A association data through the data analysis platform to obtain the preprocessed complete Q&A data / behavior Q&A association data;

[0220] The fifth storage sub-module is used to store the preprocessed complete Q&A data / behavior Q&A association data in the data warehouse;

[0221] Correspondingly, the first analysis sub-module includes:

[0222] The multi-dimensional analysis sub-module is used to perform multi-dimensional analysis on the complete Q&A data / behavior Q&A association data through a variety of data processing and analysis algorithms to generate a call analysis result.

[0223] In some embodiments, the device further includes:

[0224] The first development sub-module is used to develop a data visualization display interface for the data analysis platform through a data visualization tool;

[0225] The first output sub-module is used to output the call analysis result through the data visualization display interface.

[0226] In some embodiments, the device further includes:

[0227] A second development sub-module, configured to develop data analysis interfaces for the data analysis platform based on a web application programming interface framework and open the data analysis functions of the data analysis platform;

[0228] A third development sub-module, configured to develop a visualization interface based on a front-end framework, so that users can directly access the call analysis results in the data analysis platform through the visualization interface and the data analysis interfaces.

[0229] In some embodiments, the apparatus further includes:

[0230] A third acquisition sub-module, configured to acquire the load information of the language model service cluster;

[0231] A third generation sub-module, configured to generate an allocation policy for language model call requests according to the call analysis results and the load information, so as to process new language model call requests based on the allocation policy.

[0232] In some embodiments, the apparatus further includes:

[0233] A third monitoring sub-module, configured to monitor the health status of the language model service cluster;

[0234] A transfer sub-module, configured to transfer the language model call requests being processed in the faulty server to the remaining healthy servers for execution when a faulty server is detected in the language model service cluster.

[0235] It can be understood that for the description of the features in the embodiments corresponding to the information collection and analysis apparatus, reference can be made to the relevant descriptions in the embodiments corresponding to the information collection and analysis method, which will not be elaborated here one by one.

[0236] An embodiment of the present invention further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the embodiments of the above information collection and analysis method.

[0237] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps in any one of the embodiments of the above information collection and analysis method when running.

[0238] In an exemplary embodiment, the above computer-readable storage medium may include, but is not limited to: various media such as a USB flash drive, a read-only memory (ROM for short), a random access memory (RAM for short), a mobile hard disk, a magnetic disk, or an optical disc that can store a computer program.

[0239] An embodiment of the present invention further provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, the steps in any of the above information collection and analysis method embodiments are implemented.

[0240] An embodiment of the present invention further provides another computer program product, including a non-volatile computer-readable storage medium, where the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above information collection and analysis method embodiments are implemented.

[0241] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0242] The above has introduced in detail an information collection and analysis method, device, equipment, medium, and product provided by the present invention. Specific examples are used herein to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. An information collection and analysis method, characterized in that, Including: Monitoring language model call requests transmitted by multiple data sources through a gateway with multi - protocol recognition function and a dynamic traffic processing platform; wherein, the data sources at least include user interfaces, application programming interfaces, and integrated development environment plugins; When receiving a language model call request transmitted by a target data source, forwarding the language model call request to a language model service cluster to generate corresponding Q&A data through the language model service cluster; When the target data source is an integrated development environment plugin, forwarding the streaming Q&A data generated by the language model service cluster to the corresponding integrated development environment plugin, and aggregating the streaming Q&A data into corresponding complete Q&A data; obtaining the user behavior data generated by the integrated development environment plugin based on the corresponding streaming Q&A data; Performing full - link multi - dimensional analysis based on the Q&A data and the corresponding user behavior data to generate a call analysis result of the language model service cluster; Among them, monitoring language model call requests transmitted by multiple data sources through a gateway with multi - protocol recognition function and a dynamic traffic processing platform includes: Monitoring the traffic data transmitted by each data source through the corresponding protocol based on the gateway and the dynamic traffic processing platform; Identifying the language model call requests that conform to the language model call characteristics in each of the traffic data through deep packet inspection technology.

2. The information collection and analysis method according to claim 1, wherein When receiving a language model call request transmitted by a target data source, forwarding the language model call request to a language model service cluster includes: Generating a request fingerprint of the language model call request through a preset identifier generation algorithm; wherein, the request fingerprint at least includes metadata information such as user Internet protocol address, client type, entry type, model version, and request timestamp; Adding the request fingerprint to the request header of the language model call request; Forwarding the language model call request to the language model service cluster according to the request fingerprint in the corresponding request header.

3. The information collection and analysis method according to claim 2, wherein Forwarding the language model call request to the language model service cluster according to the request fingerprint in the corresponding request header includes: Configuring the proxy transfer instruction of the gateway and the dynamic traffic processing platform; Based on the proxy transfer instruction, controlling the gateway and the dynamic traffic processing platform to forward the language model call request to the language model service cluster according to the request fingerprint, the entry type, and the model version.

4. The information collection and analysis method according to claim 1, wherein Also including: When the target data source is a user interface or an application programming interface, forwarding the streaming Q&A data generated by the language model service cluster to the corresponding user interface or application programming interface, and aggregating the streaming Q&A data into corresponding complete Q&A data.

5. The information collection and analysis method according to claim 1, wherein Aggregating the streaming Q&A data into corresponding complete Q&A data includes: Building a streaming data processing platform, and configuring the data protocol, asynchronous processing, and concurrent parameters of the streaming data processing platform; Receiving and parsing the streaming Q&A data generated by the language model service cluster through the streaming data processing platform; Extracting each target business field in the streaming Q&A data through regular expressions; Real-time splice each of the target service fields transmitted in chunks through a content splicing algorithm, and add metadata information to each of the target service fields to generate the corresponding complete Q&A data; Among them, the metadata information at least includes a timestamp, a block sequence number, and a data source.

6. The information collection and analysis method according to claim 1, characterized in that Obtain the user behavior data generated by the integrated development environment plug-in based on the corresponding streaming Q&A data, including: Monitor the code completion event of the user through the integrated development environment plug-in; When receiving the streaming Q&A data generated by the language model service cluster through the integrated development environment plug-in, determine that the code completion event is triggered, and determine the code completion suggestion according to the streaming Q&A data; Judge whether the user accepts the code completion suggestion within a preset period; If so, generate the user behavior data indicating that the user adopts the code completion suggestion; If not, generate the user behavior data indicating that the user rejects the code completion suggestion.

7. The information collection and analysis method according to claim 6, wherein After obtaining the user behavior data generated by the integrated development environment plug-in based on the corresponding streaming Q&A data, it further includes: Build a cache space and configure the survival time of cache items in the cache space; Store a fingerprint mapping table based on the cache space; where the fingerprint mapping table contains multiple key-value pairs, the key represents a request fingerprint, and the value represents the complete Q&A data corresponding to the request fingerprint; Determine the request fingerprint of the user behavior data; Determine the key-value pair corresponding to the user behavior data in the fingerprint mapping table according to the request fingerprint of the user behavior data; Generate behavior Q&A association data according to the user behavior data and the corresponding key-value pair.

8. The information acquisition and analysis method according to claim 7, characterized in that Determine the request fingerprint of the user behavior data, including: When receiving the streaming Q&A data generated by the language model service cluster, determine the corresponding Q&A timestamp through the integrated development environment plug-in; Generate the request fingerprint of the user behavior data according to the generated user behavior data, the corresponding Q&A timestamp, and a preset identifier generation algorithm.

9. The information collection and analysis method according to claim 7, characterized in that Before performing a full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data, after obtaining the user behavior data generated by the target data source based on the Q&A data, it further includes: When the target data source is a user interface or an application programming interface, store the corresponding complete Q&A data in a pre-constructed storage space; When the target data source is an integrated development environment plug-in, store the corresponding behavior Q&A association data in the pre-constructed storage space.

10. The information collection and analysis method according to claim 9, wherein Storing the complete Q&A data or the behavior Q&A association data in the storage space includes: Build a hot data storage space, a warm data storage space, and a cold data storage space; Store the complete Q&A data or the behavior Q&A association data in the hot data storage space; Monitor the access information of the complete Q&A data or the behavior Q&A association data; Degrade and store the complete Q&A data or the behavior Q&A association data in the warm data storage space or the cold data storage space according to the access information.

11. The information collection and analysis method according to claim 10, characterized in that, Downgrading and storing the complete Q&A data or the behavior Q&A associated data into the warm data storage space or the cold data storage space according to the access information includes: Judging whether there is an access to the complete Q&A data or the behavior Q&A associated data within a first preset time according to the access information; If it is confirmed that there is an access to the complete Q&A data or the behavior Q&A associated data within the first preset time, return to the step of judging whether there is an access to the complete Q&A data or the behavior Q&A associated data within the first preset time according to the access information; If it is confirmed that there is no access to the complete Q&A data or the behavior Q&A associated data within the first preset time, downgrade and store the complete Q&A data or the behavior Q&A associated data into the warm data storage space, and judge whether there is an access to the complete Q&A data or the behavior Q&A associated data within a second preset time according to the access information; If it is confirmed that there is an access to the complete Q&A data or the behavior Q&A associated data within the second preset time, return to the step of judging whether there is an access to the complete Q&A data or the behavior Q&A associated data within the second preset time according to the access information; If it is confirmed that there is no access to the complete Q&A data or the behavior Q&A associated data within the second preset time, downgrade and store the complete Q&A data or the behavior Q&A associated data into the cold data storage space; wherein, the first preset time is less than the second preset time.

12. The information collection and analysis method according to claim 9, characterized in that Performing full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data, including: Building a data analysis platform and a corresponding data warehouse; Obtaining the complete Q&A data or the behavior Q&A associated data in the storage space and storing it into the data warehouse; Performing multi-dimensional analysis on the complete Q&A data or the behavior Q&A associated data through the data analysis platform to generate the call analysis result of the language model service cluster.

13. The information collection and analysis method according to claim 12, characterized in that, Obtaining the complete Q&A data or the behavior Q&A associated data in the storage space and storing it into the data warehouse includes: Performing data cleaning and standardization processing on the complete Q&A data or the behavior Q&A associated data through the data analysis platform to obtain the preprocessed complete Q&A data or the behavior Q&A associated data; Storing the preprocessed complete Q&A data or the behavior Q&A associated data into the data warehouse; Correspondingly, performing multi-dimensional analysis on the complete Q&A data or the behavior Q&A associated data through the data analysis platform includes: Performing multi-dimensional analysis on the complete Q&A data or the behavior Q&A associated data through a variety of data processing and analysis algorithms to generate the call analysis result.

14. The information collection and analysis method according to claim 13, characterized in that After generating the call analysis result of the language model service cluster, it further includes: Developing a data visualization display interface for the data analysis platform through a data visualization tool; Outputting the call analysis result through the data visualization display interface.

15. The information collection and analysis method according to claim 14, wherein It further includes: Develop data analysis interfaces for the data analysis platform based on a web application programming interface framework, and open the data analysis functions of the data analysis platform; Develop a visualization interface based on a front-end framework to enable users to directly access the call analysis results in the data analysis platform through the visualization interface and the data analysis interfaces.

16. The information collection and analysis method according to any one of claims 1 to 15, characterized in that, After generating the call analysis results of the language model service cluster, it further includes: Obtain the load information of the language model service cluster; Generate an allocation strategy for language model call requests based on the call analysis results and the load information, so as to process new language model call requests based on the allocation strategy.

17. The information collection and analysis method according to claim 16, wherein It further includes: Monitor the health status of the language model service cluster; When a faulty server is detected in the language model service cluster, transfer the language model call requests being processed in the faulty server to the remaining healthy servers for execution.

18. An information collection and analysis device, characterized in that, It includes: A monitoring module for monitoring the traffic data transmitted by each data source through the corresponding protocol based on a gateway and a dynamic traffic processing platform; Identify the language model call requests that conform to the language model call characteristics in each of the traffic data through deep packet inspection technology; wherein, the data sources at least include a user interface, an application programming interface, and an integrated development environment plugin; A forwarding module for forwarding the language model call requests to the language model service cluster when receiving the language model call requests transmitted by a target data source, so as to generate corresponding Q&A data through the language model service cluster; An acquisition module for, when the target data source is an integrated development environment plugin, forwarding the streaming Q&A data generated by the language model service cluster to the corresponding integrated development environment plugin, aggregating the streaming Q&A data into corresponding complete Q&A data; obtaining the user behavior data generated by the integrated development environment plugin based on the corresponding streaming Q&A data; An analysis module for performing full-link multi-dimensional analysis based on the Q&A data and the corresponding user behavior data to generate the call analysis results of the language model service cluster.

19. An electronic device, characterized in that, It includes: A memory for storing a computer program; A processor for implementing the steps of the information collection and analysis method according to any one of claims 1 to 17 when executing the computer program.

20. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the steps of the information collection and analysis method according to any one of claims 1 to 17.

21. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the steps of the information collection and analysis method according to any one of claims 1 to 17.

Images