Large model version identification method and device, electronic equipment and readable storage medium

Abstract

The invention provides a large model version recognition method and device, electronic equipment and a readable storage medium, and relates to the technical field of artificial intelligence such as large models, deep learning, natural language processing and low codes. The large model version identification method comprises the steps of obtaining a target time window according to first release time of a target version of a to-be-identified large model; obtaining a plurality of test queries according to the target time window; inserting the plurality of test queries into the service data flow of the to-be-identified large model in real time to obtain a response result output by the to-be-identified large model for each test query; obtaining an abnormal score of the to-be-identified large model according to the plurality of response results; and in response to determining that the abnormal score meets the first preset requirement, obtaining an identification result that the actual version of the to-be-identified large model is not the target version. According to the method and the device, the acquisition accuracy of test query and the identification accuracy and identification efficiency of the large model version can be improved, and the avoidance resistance during identification of the large model version is enhanced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to the technical field of artificial intelligence such as large model, deep learning, natural language processing, and low code. A large model version identification method and device, an electronic device, and a readable storage medium are provided. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large models (LM) have become the infrastructure of core technologies in various industries. In the current AI (Artificial Intelligence) service market, there are many third-party vendors providing API (Application Programming Interface) interface services of large models for users. However, since the computing cost of a high-version large model is much higher than that of a low-version large model, some vendors maliciously mix low-version large models into the API services provided to users in order to reduce operating costs and improve profit margins, forming a "model traffic cheating" phenomenon.

[0003] Such behavior not only damages the economic interests of users (users pay for high-version large models but obtain low-quality services), but also seriously affects the quality of services based on AI and the user experience. In particular, in application scenarios such as financial risk control, medical diagnosis, and legal consultation, which have extremely high accuracy requirements, the downgrade of a large model version may lead to serious business consequences. Therefore, how to accurately identify the version of a large model has become a technical problem to be solved. SUMMARY

[0004] According to a first aspect of the present disclosure, a large model version identification method is provided, comprising: obtaining a target time window according to a first release time of a target version of a large model to be identified; obtaining a plurality of test queries according to the target time window; inserting the plurality of test queries into a business data stream of the large model to be identified in real time to obtain a response result output by the large model to be identified for each test query; obtaining an abnormal score of the large model to be identified according to a plurality of response results; and obtaining an identification result that the actual version of the large model to be identified is not the target version in response to determining that the abnormal score meets a first preset requirement.

[0005] According to a second aspect of the present disclosure, an identification apparatus of a large model version is provided, comprising: a processing unit configured to obtain a target time window according to a first release time of a target version of a large model to be identified; an obtaining unit configured to obtain a plurality of test queries according to the target time window; a responding unit configured to insert the plurality of test queries into a service data stream of the large model to be identified in real time to obtain a response result output by the large model to be identified for each test query; a detecting unit configured to obtain an abnormal score of the large model to be identified according to a plurality of response results; and an identification unit configured to obtain an identification result that the actual version of the large model to be identified is not the target version in response to determining that the abnormal score meets a first preset requirement.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0007] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method described above.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method described above.

[0009] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0011] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure;

[0012] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure;

[0014] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure;

[0015] Figure 5is a schematic diagram according to a fifth embodiment of the present disclosure;

[0016] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0017] Figure 7 is a schematic diagram according to a seventh embodiment of the present disclosure;

[0018] Figure 8 is a schematic diagram according to an eighth embodiment of the present disclosure;

[0019] Figure 9 is a block diagram of an electronic device for implementing the identification method of the large model version according to the embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, descriptions of well-known functions and mechanisms are omitted in the following description.

[0021] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure. As shown in Figure 1 , the identification method of the large model version according to the present embodiment specifically includes the following steps:

[0022] S101, obtaining a target time window according to a first release time of a target version of a large model to be identified;

[0023] S102, obtaining a plurality of test queries according to the target time window;

[0024] S103, inserting the plurality of test queries into a business data stream of the large model to be identified in real time to obtain a response result output by the large model to be identified for each test query;

[0025] S104, obtaining an anomaly score of the large model to be identified according to a plurality of response results;

[0026] S105, in response to determining that the anomaly score meets a first preset requirement, obtaining an identification result that the actual version of the large model to be identified is not the target version.

[0027] The identification method of the large model version of the embodiment can obtain a test query for identifying the to-be-identified large model according to a target time window obtained according to the first release time of the target version of the to-be-identified large model, so that the obtained test query is closely related to the release time of the target version of the to-be-identified large model, and the accuracy of the test query and the accuracy of the identification of the large model version can be improved. On the other hand, the response result for obtaining the abnormal score of the to-be-identified large model is obtained by inserting the test query into the business data stream of the to-be-identified large model in real time, so that the purpose of real-time identification of the large model version is achieved, and the identification efficiency of the large model version and the anti-evasion capability during identification of the large model version are improved.

[0028] In the embodiment, the large model can be a large language model (LLM) or a multimodal large model (MLM).

[0029] In the embodiment, the target version of the to-be-identified large model can be the latest version of the to-be-identified large model or the online deployment version of the to-be-identified large model at the current time.

[0030] When S101 is performed, the target version of the to-be-identified large model can be determined first, then the first release time of the to-be-identified large model is obtained according to the target version, and finally the target time window is obtained according to the obtained first release time.

[0031] Specifically, when the target time window is obtained according to the first release time of the target version of the to-be-identified large model in S101, the following implementation manner can also be used: according to the target version of the to-be-identified large model, a historical version of the to-be-identified large model is determined, and a second release time of the historical version is obtained. The historical version in the embodiment is a version before the target version, and the second release time is earlier than the first release time; according to the first release time and the second release time, a first knowledge cutoff time and a second knowledge cutoff time of the to-be-identified large model are determined. The relationship between the knowledge cutoff time and the release time in the embodiment is “first release time>first knowledge cutoff time>second release time>second knowledge cutoff time”, that is, the first release time is the latest, and the second knowledge cutoff time is the earliest; and the target time window is obtained according to the first knowledge cutoff time and the second knowledge cutoff time.

[0032] That is, the embodiment utilizes the knowledge cutoff date of different versions of large models to accurately define the target time window, so that the test query obtained based on the target time window can meet the requirement that the new version of large models should be able to accurately answer the events occurring within the training cutoff time, while the old version of large models cannot accurately answer or has knowledge gaps, thereby achieving the purpose of identifying the version of large models according to the difference in the knowledge cutoff time of different versions of large models.

[0033] The historical version determined in S101 can be any version before the target version of the to-be-identified large model, or a version manually input or selected by a user.

[0034] In the embodiment, when S101 obtains the target time window according to the first knowledge cutoff time and the second knowledge cutoff time, the implementation manner can be as follows: obtaining a time buffer parameter according to the model type of the to-be-identified large model, the embodiment can obtain the time buffer parameter corresponding to the to-be-identified large model according to the preset correspondence between the model type and the time buffer parameter; and obtaining the target time window according to the first knowledge cutoff time, the second knowledge cutoff time and the time buffer parameter.

[0035] For a large model that is updated more frequently, the embodiment can set a smaller time buffer parameter, for example, 7 days, and for a large model that is not updated frequently, the embodiment can set a larger time buffer parameter, for example, 15 days.

[0036] The time buffer parameter in the embodiment is used to avoid the boundary effect of the large model training, so as to ensure that the test query obtained according to the target time window is within the knowledge boundary of the to-be-identified large model.

[0037] For example, if the first knowledge cutoff time is TC1, if the second knowledge cutoff time is TC2, and if the existing time buffer parameter is , the target time window obtained by the embodiment in S101 can be ; if there is no time buffer parameter, the target time window obtained by the embodiment in S101 can be .

[0038] In the embodiment, when S101 is performed, the release time of different versions of the to-be-identified large model can be obtained through different information channels such as official API (Application Programming Interface, application programming interface) document and release announcement.

[0039] To further ensure the reliability of the target time window, after obtaining the publication time and the knowledge cutoff time in S101, the embodiment can further perform multi-source cross verification on the obtained time information, and then perform the step of obtaining the target time window according to the knowledge cutoff time after verification.

[0040] In addition, when obtaining the target time window according to the first publication time of the target version in S101, the embodiment can also not obtain the knowledge cutoff time, and directly take a historical period before the first publication time as the target time window.

[0041] After obtaining the target time window in S101, the embodiment performs S102 to obtain a plurality of test queries according to the obtained target time window; wherein the test queries obtained by the embodiment in S102 are input to the to-be-identified large model for version identification.

[0042] When performing S102, the embodiment can obtain a plurality of queries corresponding to the target time window from a preset query library as a plurality of test queries; the query library of the embodiment includes a plurality of times and queries corresponding to different times, and can further include standard answers corresponding to different queries.

[0043] That is, the embodiment can obtain a plurality of queries in the preset query library that match the target time window as test queries for version identification of the to-be-identified large model, which can simplify the acquisition step of the test query and improve the acquisition efficiency of the test query.

[0044] After obtaining a plurality of test queries in S102, the embodiment performs S103 to insert the plurality of test queries into the business data stream of the to-be-identified large model in real time to obtain the response result output by the to-be-identified large model for each test query.

[0045] That is, the embodiment inserts the obtained test query as a normally input query of the user into the business data stream of the to-be-identified large model in real time to obtain the response result of the corresponding test query output by the to-be-identified large model, so as to achieve the purpose of identifying the version of the to-be-identified large model according to the obtained response result.

[0046] Specifically, when performing S103 to insert the plurality of test queries into the business data stream of the to-be-identified large model in real time, the implementation manner that can be adopted by the embodiment is as follows: obtaining a historical query request frequency of the to-be-identified large model, the historical query request frequency obtained by the embodiment being a query request frequency corresponding to a historical moment, the historical moment can be minutes, hours, days or weeks before the target moment, etc.; obtaining a predicted query request frequency corresponding to the target moment according to the obtained historical query request frequency; obtaining a target insertion frequency corresponding to the target moment according to the obtained predicted query request frequency and a preset insertion weight value, the target insertion frequency obtained by the embodiment being the number of test queries to be inserted in a unit time, the multiplication result between the predicted query request frequency and the preset insertion weight value can be taken as the target insertion frequency, and the preset insertion weight value in the embodiment can be 0.01, etc.; and inserting the plurality of test queries into the business data stream of the to-be-identified large model in real time at the target moment according to the target insertion frequency.

[0047] That is, the embodiment inserts the test queries obtained according to the target time window into the business data stream of the to-be-identified large model in real time according to the obtained target insertion frequency, so that it is difficult for the supplier of the to-be-identified large model to identify the test queries in the business data stream for version identification, thereby effectively preventing the supplier of the to-be-identified large model from performing targeted avoidance and improving the accuracy of the response result returned by the to-be-identified large model for the test query.

[0048] The target moment in the embodiment can be one (for example, the current moment) or multiple (for example, the current moment and at least one future moment); that is, when performing S103, the embodiment can insert all the test queries into the business data stream at one target moment, or can insert the test queries into the business data stream in batches at multiple target moments.

[0049] In addition, before performing S103 to insert the plurality of test queries into the business data stream of the to-be-identified large model in real time, the embodiment can further include the following content: performing disguise processing on the plurality of test queries to obtain query disguise results corresponding to the plurality of test queries respectively; and inserting the plurality of query disguise results into the business data stream of the to-be-identified large model in real time.

[0050] That is, the embodiment can further perform disguise processing on the test queries inserted into the business data stream of the to-be-identified large model, so that the test queries inserted into the business data stream of the to-be-identified large model are more similar to real queries and are more difficult to be detected, thereby further avoiding the supplier of the to-be-identified large model from identifying the test queries and performing targeted avoidance.

[0051] In the embodiment, when the test query is disguised in S103, the test query can be format disguised, that is, the data format of the test query is converted into a specific data format (for example, JSON format); or the test query can be semantically disguised, that is, according to the words and sequences commonly used in normal queries, the test query is replaced by synonyms, the sequence is adjusted, etc.

[0052] After obtaining the disguised result of the test query in S103, the embodiment can further obtain a disguise score of the disguised result, for example, by a large model to obtain the disguise score of the disguised result, and then in a case where the obtained disguise score is greater than a second preset score threshold, the disguised result is inserted into the business data stream of the to-be-identified large model in real time; otherwise, the test query is re-disguised until the obtained disguise score is greater than the second preset score threshold.

[0053] The response result obtained in S103 of the embodiment is an answer output by the to-be-identified large model in real-time response to the test query.

[0054] After obtaining the response result output by the to-be-identified large model for each test query in S103, the embodiment performs S104 to obtain an abnormal score of the to-be-identified large model according to the multiple response results.

[0055] In S104, the detection score of the to-be-identified large model corresponding to the preset dimension can be obtained according to the multiple response results, and then the abnormal score of the to-be-identified large model is obtained according to the obtained detection score corresponding to the preset dimension; the preset dimension in the embodiment can be at least one of time dimension, space dimension and behavior dimension.

[0056] In S104, the abnormal score of the to-be-identified large model is obtained according to the multiple response results, which can first obtain a first detection score according to the response time corresponding to the multiple response results, and then obtain the abnormal score of the to-be-identified large model according to the first detection score; the response time is the time when the to-be-identified large model returns the response result corresponding to the test query; that is, the to-be-identified large model can be detected in the time dimension, so as to obtain the abnormal score according to the first detection score corresponding to the time dimension.

[0057] In S104, the first detection score is obtained according to the multiple response times, which can obtain the mean value of the response time according to the multiple response times, and then take the score corresponding to the mean value of the response time as the first detection score; the embodiment will preset the score corresponding to different mean values of the response time.

[0058] In the implementation example, when the abnormal score of the large model to be identified is obtained according to the plurality of response results in S104, the standard answer corresponding to each test query can be obtained first, and then the first word embedding vector corresponding to the response result and the second word embedding vector corresponding to the standard answer are obtained. The second detection score is obtained according to the first word embedding vector and the second word embedding vector corresponding to the same test query (for example, the distance or similarity between the two word embedding vectors is converted into a score), and finally the abnormal score of the large model to be identified is obtained according to the second detection score. That is, the large model to be identified can be detected in the spatial dimension (that is, the feature space), so as to obtain the abnormal score according to the second detection score corresponding to the spatial dimension.

[0059] In the implementation example, when the abnormal score of the large model to be identified is obtained according to the plurality of response results in S104, the standard answer corresponding to each test query can be obtained first, and then the first word embedding vector corresponding to the response result and the second word embedding vector corresponding to the standard answer are obtained. The second detection score is obtained according to the first word embedding vector and the second word embedding vector corresponding to the same test query (for example, the distance or similarity between the two word embedding vectors is converted into a score), and finally the abnormal score of the large model to be identified is obtained according to the second detection score. That is, the large model to be identified can be detected in the spatial dimension (that is, the feature space), so as to obtain the abnormal score according to the second detection score corresponding to the spatial dimension.

[0060] It can be understood that, when S104 is performed, the abnormal score of the large model to be identified can also be obtained according to the detection scores and weight coefficients corresponding to different detection dimensions.

[0061] Specifically, when S104 is performed, the abnormal score of the large model to be identified can be obtained by using the following calculation formula:

[0062]

[0063] In the above calculation formula: represents the abnormal score of the large model to be identified; 、 、 respectively represent the first detection score corresponding to the time dimension, the second detection score corresponding to the spatial dimension, and the third detection score corresponding to the behavior dimension; 、 and respectively represent the weight coefficient corresponding to the time dimension, the weight coefficient corresponding to the spatial dimension, and the weight coefficient corresponding to the behavior dimension.

[0064] After obtaining the abnormal score of the to-be-identified large model in S104, the embodiment performs S105 to obtain the identification result that the actual version of the to-be-identified large model is not the target version in response to determining that the abnormal score meets the first preset requirement.

[0065] When performing S105, the embodiment can determine that the abnormal score meets the first preset requirement in a case where the abnormal score is a preset score value (i.e., the abnormal score is 1), thereby obtaining the identification result that the actual version of the to-be-identified large model is not the target version.

[0066] When performing S105, the embodiment can first obtain an abnormal level of the to-be-identified large model according to the abnormal score, and then determine that the abnormal score meets the first preset requirement in response to determining that the abnormal level is a preset level, thereby obtaining the identification result that the actual version of the to-be-identified large model is not the target version.

[0067] The embodiment can preset a correspondence between the abnormal level and the abnormal score, for example, the abnormal score corresponding to normal can be 0-0.3, the abnormal score corresponding to a slight abnormal level can be 0.3-0.6, the abnormal score corresponding to a moderate abnormal level can be 0.6-0.8, and the abnormal score corresponding to a serious abnormal level can be 0.8-1. Therefore, when performing S105, the embodiment determines the abnormal level corresponding to the abnormal score according to the preset correspondence.

[0068] After obtaining the identification result that the actual version of the to-be-identified large model is not the target version in S105, the embodiment can further trigger an alarm mechanism to generate alarm information. The alarm information generated by the embodiment can be a report corresponding to the identification result, or can include a processing suggestion corresponding to the identification result.

[0069] After generating the alarm information in S105, the embodiment can send the generated alarm information to the corresponding personnel in real time through a communication mode such as an email or a short message.

[0070] That is, the embodiment can determine that the to-be-identified large model has the phenomenon of "model traffic cheating" in a case where the abnormal score meets the first preset requirement. More specifically, the supplier of the to-be-identified large model has mixed a low-version large model into the service provided by the supplier, so that the actual version of the to-be-identified large model includes other versions in addition to the target version.

[0071] Figure 2 is a schematic diagram according to the second embodiment of the present disclosure. As shown in Figure 2 When performing S102 "obtaining a plurality of test queries according to the target time window", the embodiment can adopt the following implementation manner:

[0072] S201, acquire a plurality of events with occurrence time located in the target time window as a plurality of candidate events;

[0073] S202, respectively, quality evaluation is carried out on the plurality of candidate events, and the quality score of each candidate event is obtained;

[0074] S203, select a plurality of candidate events whose quality scores meet the second preset requirement as a plurality of target events;

[0075] S204, according to the plurality of target events, the plurality of test queries are obtained.

[0076] That is, the embodiment can obtain the test query in real time through the determined target time window, and then use the test query obtained in real time to identify the version of the to-be-identified large model, which is more difficult to be detected by the supplier of the to-be-identified large model than using the existing test set, thereby improving the accuracy and effectiveness of the obtained test query.

[0077] The candidate events obtained by the embodiment in S201 can be different types of factual events such as news, technological development, society, culture and sports obtained from the Internet and located in the target time window.

[0078] When the embodiment performs S202 to evaluate the quality of the candidate events and obtain the quality score of the candidate events, it can first obtain the first score corresponding to the candidate event (the first score is the media coverage score, which can be determined based on the number of reports and the authority of the media, the more the number of reports or the higher the authority of the media, the higher the first score), the second score (the second score is the social influence score, which can be determined based on the social media discussion heat of the event, the higher the social media discussion heat, the higher the second score) and the third score (the third score is the uniqueness score, which can be determined based on whether the event is located in the existing test set, if the candidate event is not located in the existing test set, the third score is 1, otherwise the third score is 0), then according to the score of each dimension and the weight coefficient of each dimension, the importance score of the candidate event is obtained, and finally the importance score obtained is taken as the quality score of the candidate event.

[0079] Specifically, the embodiment can obtain the importance score of the candidate event using the following calculation formula when performing S202:

[0080] In the above calculation formula: represents the importance score of the candidate event e; represents the media coverage score of the candidate event e, represents the social influence score of the candidate event e, represents the uniqueness score of the candidate event e; 、 With are weight coefficients corresponding to different dimensions, which can be 0.5, 0.3 and 0.2 respectively.

[0081] It can be understood that, in order to further improve the accuracy of the selected target event, the embodiment can further perform verifiability evaluation and selective evaluation on the candidate event before performing S202 to calculate the importance score of the candidate event, and then perform the calculation of the importance score of the candidate event after completing the verifiability evaluation and the selective evaluation.

[0082] Among them, when the embodiment performs S202 to evaluate the verifiability of the candidate event, it can be determined that the candidate event passes the verifiability evaluation when it is determined that the candidate event has a clear answer; when the embodiment performs S202 to perform selective evaluation on the candidate event, it can be determined that the candidate event passes the selective evaluation when it is determined that the candidate event has a large model distinguishability (i.e. the candidate event can effectively distinguish large models with different knowledge); the embodiment can evaluate the verifiability and selectivity of the candidate event through artificial evaluation or large model evaluation.

[0083] When the embodiment performs S203 to select a candidate event with a quality score meeting a second preset requirement as a target event, it can select a candidate event with a quality score greater than a first preset score threshold as a target event, or select a candidate event ranked in the top N as a target event, N is a positive integer greater than 1.

[0084] Preferably, the value range of N in the embodiment can be 50-200, and the value of N can be adjusted according to the actual detection accuracy requirement.

[0085] When the embodiment performs S204 to obtain a plurality of test queries according to a plurality of selected target events, it can generate a question corresponding to each target event according to the event content of the target event, and then use the generated question as a test query corresponding to the target event.

[0086] When the embodiment performs S204, it can generate a question corresponding to a target event through artificial annotation, or use a large model to generate a question corresponding to a target event; wherein the type of the question generated by the embodiment can include at least one of direct questioning type, indirect reasoning type and comparison analysis type.

[0087] In addition, after the embodiment performs S204 to generate questions corresponding to different target events, it can also establish a difficulty grading system for the questions, i.e. dividing the generated questions into basic level questions, intermediate level questions or advanced level questions according to the difficulty of the questions.

[0088] In addition, after obtaining the plurality of test queries in S204, the embodiment can further include the following: for each test query, inputting the test query into the plurality of candidate large models respectively, obtaining a plurality of candidate answers according to the output results of the plurality of candidate large models; clustering the plurality of candidate answers according to the semantics of the candidate answers to obtain at least one answer set; and selecting a candidate answer from the answer set including the most candidate answers as the standard answer corresponding to the test query.

[0089] That is, after obtaining the test query corresponding to the target event, the embodiment uses the known high-version large model (i.e., the candidate large model) to complete the acquisition of the standard answer through cross-validation, which can effectively improve the accuracy of the obtained standard answer and reduce the acquisition cost of the standard answer.

[0090] In the embodiment, when selecting a candidate answer from the answer set including the most candidate answers in S204, the candidate answer with the highest confidence in the answer set including the most candidate answers can be selected as the standard answer corresponding to the test query.

[0091] After obtaining the test query and the standard answer corresponding to the target event in S204, the embodiment can store the test query and the standard answer corresponding to the test query in a query library according to the occurrence time of the target event. The embodiment can also perform quality control on the queries and the standard answers corresponding to the queries in the query library, such as repeated detection, step-by-step update, version management of the query library, and the like.

[0092] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure. As shown in Figure 3 When performing S104 "obtaining an abnormal score of the large model to be identified according to the plurality of response results", the embodiment can use the following implementation manner:

[0093] S301, obtaining a plurality of standard answers according to the test queries corresponding to the plurality of response results respectively;

[0094] S302, obtaining query accuracies corresponding to different event types according to the response results and the standard answers of the test queries corresponding to different event types;

[0095] S303, obtaining a compliance degree score of the large model to be identified according to the query accuracies corresponding to different event types and a weight coefficient;

[0096] S304, in response to determining that the compliance degree score meets a third preset requirement, obtaining a preset score as the abnormal score of the large model to be identified.

[0097] That is, the embodiment obtains the compliance degree score of the to-be-identified large model according to the query accuracy and the weight coefficient corresponding to different event types, and then obtains the anomaly score of the to-be-identified large model according to the compliance degree score, so that in the process of obtaining the anomaly score of the to-be-identified large model, the information of the event type corresponding to the test query is combined, and the accuracy of the obtained anomaly score can be further improved.

[0098] The event type in the embodiment is the type of the target event of the test query, for example, a news event type, a technology development event type, a culture and sports event type, etc.

[0099] When the embodiment performs S302 to obtain the query accuracy corresponding to different event types, the query accuracy corresponding to each event type can be obtained according to the response result and the standard answer of the test query corresponding to the event type, for example, the ratio between the number of response results similar to the standard answer and the total number of response results is taken as the query accuracy.

[0100] When the embodiment performs S303, the compliance degree score of the to-be-identified large model can be obtained using the following calculation formula:

[0101]

[0102] In the above calculation formula: represents the compliance degree score of the to-be-identified large model s at time t; represents the weight coefficient of the i-th event type; represents the accuracy of the to-be-identified large model s at time t for the i-th event type.

[0103] When the embodiment performs S304, the anomaly score of the to-be-identified large model can be obtained using the following calculation formula:

[0104]

[0105] In the above calculation formula: represents the anomaly score of the to-be-identified large model s at time t; represents the compliance degree score of the to-be-identified large model s at time t; represents the anomaly detection sensitivity parameter; represents the historical compliance degree score of the to-be-identified large model s.

[0106] That is, the embodiment determines that the compliance degree score of the to-be-identified large model is less than the product of the historical compliance degree score of the to-be-identified large model and the anomaly detection sensitivity parameter, that is, the compliance degree score meets the third preset requirement, and takes the preset score 1 as the anomaly score of the to-be-identified large model.

[0107] Figure 4 is a schematic diagram according to the fourth embodiment of the present disclosure. Figure 4 A flowchart of the present embodiment when obtaining test queries is shown: S401, obtaining a target time window according to a target version and a historical version of a large model to be identified; S402, obtaining a plurality of candidate events corresponding to different event types from the Internet according to the obtained target time window; S403, performing quality evaluation on the obtained plurality of candidate events, and selecting candidate events ranked in the top N in terms of quality scores as target events; S404, generating questions corresponding to each target event, and taking the generated questions as test queries; S405, generating standard answers corresponding to different test queries; and S406, storing the obtained test queries and corresponding standard answers to a query library.

[0108] That is, in the present embodiment, when identifying a large model version, the test queries used for version identification are obtained in real time through the obtained target time window, which can improve the accuracy of the obtained test queries, and compared with using queries in an existing test set for identification, the anti-evasion capability when identifying a large model version can be further enhanced.

[0109] In addition, after the obtained test queries are stored in the query library, the queries included in the query library can be used for version identification of other large models.

[0110] Figure 5 is a schematic diagram according to the fifth embodiment of the present disclosure. Figure 5 An alarm flowchart of the present embodiment is shown: S501, obtaining a plurality of response results and a plurality of standard answers corresponding to the plurality of response results; S502, performing multi-dimensional anomaly detection according to the plurality of response results and the plurality of standard answers to obtain a first detection score corresponding to a time dimension, a second detection score corresponding to a space dimension, and a third detection score corresponding to a behavior dimension; S503, obtaining an anomaly score of a large model to be identified according to the first detection score, the second detection score, and the third detection score; S504, determining an anomaly level of the large model to be identified according to the anomaly score; S505, selecting an alarm strategy according to the anomaly level, the alarm strategy in the present embodiment including hierarchical alarm, adaptive alarm, etc., and then generating alarm information according to the selected alarm strategy; and S506, outputting the alarm information through a plurality of channels, for example, sending the alarm information to corresponding personnel, generating a report according to the alarm information, generating a processing suggestion (such as automatic processing or manual review) according to the alarm information, etc.

[0111] In addition, the present embodiment can also perform optimization of machine learning according to feedback after the alarm, for example, adjusting the weight coefficients of different detection scores.

[0112] That is, the embodiment realizes the alarm after the identification of the to-be-identified large model version is completed by performing multi-dimensional anomaly detection, can realize automatic detection of large model anomalies, and greatly reduces operating costs.

[0113] Figure 6 is a schematic diagram according to the sixth embodiment of the disclosure. Figure 6 Another alarm flowchart of the embodiment is shown; S601, performing camouflage processing on the test query to obtain a query camouflage result; S602, inserting the query camouflage result into the business data stream of the to-be-identified large model in real time according to the target insertion frequency; S603, obtaining a plurality of response results returned by the to-be-identified large model; S604, obtaining query accuracies corresponding to different event types according to the plurality of response results and the standard answers corresponding to the plurality of response results respectively; S605, obtaining a compliance degree score of the to-be-identified large model according to the query accuracies corresponding to different event types and the weight coefficients; S606, obtaining an anomaly score of the to-be-identified large model according to the compliance degree score, if the compliance degree score meets the third preset requirement, the anomaly score is 1, otherwise it is 0; S607, in response to determining that the anomaly score is 1, triggering an alarm mechanism to generate alarm information.

[0114] That is, the embodiment obtains the anomaly score of the to-be-identified large model according to the query accuracies corresponding to different event types, and then determines whether to alarm according to the obtained anomaly score, can realize automatic detection of large model anomalies, and greatly reduces operating costs.

[0115] Figure 7 is a schematic diagram according to the seventh embodiment of the disclosure. Figure 7 The overall architecture diagram of the large model version identification system of the embodiment is shown, including an access layer, a business logic layer, a data processing layer, a data storage layer, an external integration layer, and a security and operation layer.

[0116] Among them, the access layer includes API Gateway (API gateway), Web Dashboard (Web dashboard), SDK (software development kit) integration package, Webhook (network hook) interface, etc.

[0117] The business logic layer includes a fact extraction module, a verification module, a detection module and an analysis module; wherein the fact extraction module includes a data acquisition submodule (for concurrent calling of multi-source API, supporting flow limiting and retry mechanism), a data cleaning submodule (for deduplication, data verification, formatting processing), an event recognition submodule (for identifying key events and entities using natural language processing technology), a quality evaluation submodule (for fact quality scoring based on multi-dimensional indicators), a storage management submodule (for efficient indexing and query optimization); the verification module includes a model management submodule (for unified management of reference model calling and configuration), a query execution submodule (for efficient concurrent query distribution and management), a response analysis submodule (for semantic analysis and consistency checking), a consensus generation submodule (for majority voting and confidence weighting algorithm), an answer library management (for version control and update mechanism of standard answers); the detection module includes a traffic analysis submodule (for real-time business traffic pattern recognition and analysis), a query camouflage submodule (for intelligent query rewriting and camouflage generation), an insertion strategy submodule (for dynamic insertion timing and frequency control), a real-time monitoring submodule (for low-latency response quality detection and evaluation), a data collection submodule (for high-throughput data collection and caching); the analysis module includes a feature extraction submodule (for multi-dimensional data feature extraction and selection), an anomaly detection submodule (for integration and tuning of multiple machine learning algorithms), a baseline management submodule (for dynamic baseline calculation and update), an alarm strategy submodule (for flexible alarm rules and hierarchical management), a prediction model submodule (for trend prediction based on historical data).

[0118] The data processing layer is used for real-time stream processing, batch processing, message queue management, distributed computing, etc.

[0119] The data storage layer includes a relational database, a document database, a time series database, a distributed file system, a cache system, etc.

[0120] The external integration layer includes news API, academic data API, AI model API, third-party notification service, etc.

[0121] The security and operation layer is used for identity authentication, data encryption, log management, monitoring and alarm, disaster recovery backup, etc.

[0122] The overall identification process of the large model version identification system in the embodiment is: 1) user demand analysis; 2) system configuration initialization; 3) automatic extraction of time-sensitive facts; 4) multi-model cross-validation; 5) construction of standard answer library; 6) business traffic analysis modeling; 7) dynamic detection query insertion; 8) real-time response quality analysis; 9) multi-dimensional anomaly detection; 10) intelligent alarm and processing suggestion; 11) result output and feedback optimization.

[0123] The large model version identification system in the embodiment can be applied to different scenarios such as an AI service quality monitoring platform, an enterprise-level AI procurement decision support system, and a regulatory agency AI service compliance audit system, to achieve the purposes of improving identification accuracy, performing real-time monitoring, reducing operating costs, enhancing anti-avoidance capability, and providing quantitative evaluation.

[0124] Figure 8 is a schematic diagram according to an eighth embodiment of the present disclosure. As shown in Figure 8 The large model version identification apparatus 800 of the embodiment includes:

[0125] The processing unit 801 obtains a target time window according to a first release time of a target version of a to-be-identified large model.

[0126] The acquisition unit 802 acquires a plurality of test queries according to the target time window.

[0127] The response unit 803 inserts the plurality of test queries into a business data stream of the to-be-identified large model in real time to obtain response results output by the to-be-identified large model for each test query.

[0128] The detection unit 804 obtains an abnormal score of the to-be-identified large model according to the plurality of response results.

[0129] The identification unit 805 obtains an identification result that the actual version of the to-be-identified large model is not the target version in response to determining that the abnormal score meets a first preset requirement.

[0130] In the embodiment, the large model can be a large language model (LLM) or a multimodal large model (MLM).

[0131] In the embodiment, the target version of the to-be-identified large model can be the latest version of the to-be-identified large model or an online deployment version of the to-be-identified large model at the current time.

[0132] The processing unit 801 can first determine the target version of the to-be-identified large model, then acquire the first release time of the to-be-identified large model according to the target version, and finally obtain the target time window according to the acquired first release time.

[0133] Specifically, when obtaining the target time window according to the first release time of the target version of the large model to be identified, the processing unit 801 can further adopt the following implementation manner: determining a historical version of the large model to be identified according to the target version of the large model to be identified, and obtaining a second release time of the historical version; determining a first knowledge cutoff time and a second knowledge cutoff time of the large model to be identified according to the first release time and the second release time; and obtaining the target time window according to the first knowledge cutoff time and the second knowledge cutoff time.

[0134] That is, the processing unit 801 accurately defines the target time window by using the knowledge cutoff times of different versions of the large model, so that the test queries obtained based on the target time window can meet the requirement that the new version of the large model should be able to accurately answer the events occurring within its training cutoff time, while the old version of the large model cannot accurately answer or has knowledge gaps, thereby achieving the purpose of version identification according to the differences between different versions of the large model in the knowledge cutoff time.

[0135] The historical version determined by the processing unit 801 can be any version before the target version of the large model to be identified, or a version manually input or selected by a user.

[0136] When obtaining the target time window according to the first knowledge cutoff time and the second knowledge cutoff time, the processing unit 801 can adopt the following implementation manner: obtaining a time buffer parameter according to the model type of the large model to be identified; and obtaining the target time window according to the first knowledge cutoff time, the second knowledge cutoff time, and the time buffer parameter.

[0137] The time buffer parameter in this embodiment is used to avoid the boundary effect of the large model training, so as to ensure that the test queries obtained according to the target time window are within the knowledge boundary of the large model to be identified.

[0138] The processing unit 801 can obtain the release times of different versions of the large model to be identified through different information channels such as official API (Application Programming Interface, application programming interface) document, release announcement, etc.

[0139] In order to further ensure the reliability of the target time window, after obtaining the release time and the knowledge cutoff time, the processing unit 801 can further perform multi-source cross verification on the obtained time information, and then perform the step of obtaining the target time window according to the knowledge cutoff time after verification.

[0140] In addition, the processing unit 801 can also not perform the acquisition of the knowledge deadline when obtaining the target time window according to the first release time of the target version, and directly take a historical period before the first release time as the target time window.

[0141] After the processing unit 801 obtains the target time window, the acquisition unit 802 acquires a plurality of test queries according to the obtained target time window; wherein the test queries acquired by the acquisition unit 802 are input into the to-be-identified large model for version identification.

[0142] The acquisition unit 802 can acquire a plurality of queries corresponding to the target time window from a preset query library as the plurality of test queries; the query library of the embodiment includes a plurality of times and queries corresponding to different times, and can further include standard answers corresponding to different queries.

[0143] That is, the acquisition unit 802 can acquire a plurality of queries in the preset query library that match the target time window as test queries for version identification of the to-be-identified large model, which can simplify the acquisition step of the test queries and improve the acquisition efficiency of the test queries.

[0144] The acquisition unit 802 can also use the following implementation manner when acquiring the plurality of test queries according to the target time window: acquiring a plurality of events whose occurrence times are within the target time window as a plurality of candidate events; respectively performing quality evaluation on the plurality of candidate events to obtain a quality score of each candidate event; selecting a plurality of candidate events whose quality scores satisfy a second preset requirement as a plurality of target events; and obtaining the plurality of test queries according to the plurality of target events.

[0145] That is, the acquisition unit 802 can perform real-time acquisition of the test queries through the determined target time window, and then use the real-time acquired test queries to identify the version of the to-be-identified large model, which is less likely to be detected by the supplier of the to-be-identified large model compared with using an existing test set, thereby improving the accuracy and effectiveness of the obtained test queries.

[0146] When the acquisition unit 802 performs quality evaluation on the candidate events to obtain the quality score of the candidate event, the acquisition unit 802 can first acquire a first score, a second score and a third score corresponding to the candidate event, then obtain an importance score of the candidate event according to the scores of each dimension and the weight coefficients of each dimension, and finally take the obtained importance score as the quality score of the candidate event.

[0147] It can be understood that, in order to further improve the accuracy of the selected target event, the acquisition unit 802 can also perform verifiability evaluation and selective evaluation on the candidate event before calculating the importance score of the candidate event, and then calculate the importance score of the candidate event after completing the verifiability evaluation and selective evaluation.

[0148] When the acquisition unit 802 selects a candidate event with a quality score meeting the second preset requirement as a target event, the candidate event with a quality score greater than a first preset score threshold can be selected as a target event, or the candidate event ranked in the top N positions can be selected as a target event, where N is a positive integer greater than 1.

[0149] Preferably, the value range of N in the embodiment can be 50-200, and the value of N can be adjusted according to the actual detection accuracy requirement.

[0150] When the acquisition unit 802 obtains a plurality of test queries according to the plurality of selected target events, for each target event, a question corresponding to the target event can be generated according to the event content of the target event, and then the generated question is used as a test query corresponding to the target event.

[0151] In addition, after obtaining a plurality of test queries, the acquisition unit 802 can further include the following content: for each test query, the test query is input into a plurality of candidate large models respectively, and a plurality of candidate answers are obtained according to the output results of the plurality of candidate large models; a plurality of candidate answers are clustered according to the semantics of the candidate answers to obtain at least one answer set; and a candidate answer is selected from an answer set including the most candidate answers as a standard answer corresponding to the test query.

[0152] That is, after obtaining the test query corresponding to the target event, the acquisition unit 802 uses the known high-version large model (i.e., the candidate large model) to complete the acquisition of the standard answer through cross-validation, which can effectively improve the accuracy of the obtained standard answer and reduce the acquisition cost of the standard answer.

[0153] After obtaining the test query and the standard answer corresponding to the target event, the acquisition unit 802 can also store the test query and the corresponding standard answer into a query library according to the occurrence time of the target event; the embodiment can also perform quality control on the queries and the corresponding standard answers included in the query library, such as repeated detection, step-by-step update of the queries, version management of the query library, etc.

[0154] After the plurality of test queries are acquired by the acquisition unit 802, the plurality of test queries are inserted into the service data stream of the to-be-identified large model in real time by the response unit 803, and response results output by the to-be-identified large model for each test query are obtained.

[0155] That is, the response unit 803 inserts the acquired test queries as normally input queries of a user into the service data stream of the to-be-identified large model in real time to acquire response results of the to-be-identified large model corresponding to the test queries, so as to achieve the purpose of identifying the version of the to-be-identified large model according to the acquired response results.

[0156] Specifically, when the plurality of test queries are inserted into the service data stream of the to-be-identified large model in real time, the response unit 803 can adopt an implementation manner that: a historical query request frequency of the to-be-identified large model is acquired; a predicted query request frequency corresponding to a target time is obtained according to the acquired historical query request frequency; a target insertion frequency corresponding to the target time is obtained according to the obtained predicted query request frequency and a preset insertion weight value; and the plurality of test queries are inserted into the service data stream of the to-be-identified large model in real time at the target time according to the target insertion frequency.

[0157] That is, the response unit 803 inserts the test queries acquired according to the target time window into the service data stream of the to-be-identified large model in real time according to the obtained target insertion frequency, so that the supplier of the to-be-identified large model cannot identify the test queries for version identification in the service data stream, thereby effectively preventing the supplier of the to-be-identified large model from performing targeted avoidance, and improving the accuracy of the response results returned by the to-be-identified large model for the test queries.

[0158] The target time in the embodiment can be one (for example, the current time) or multiple (for example, the current time and at least one future time); that is, the embodiment can insert all the plurality of test queries into the service data stream at one target time, or can insert the plurality of test queries into the service data stream in batches at multiple target times.

[0159] In addition, before the plurality of test queries are inserted into the service data stream of the to-be-identified large model in real time, the response unit 803 can further include the following content: performing a disguise process on the plurality of test queries to obtain a plurality of query disguise results corresponding to the plurality of test queries, respectively; and inserting the plurality of query disguise results into the service data stream of the to-be-identified large model in real time.

[0160] That is, the response unit 803 can further perform camouflage processing on the test query inserted into the business data stream of the to-be-identified large model, so that the test query inserted into the business data stream of the to-be-identified large model is more similar to the real query and is more difficult to be detected, further avoiding the supplier of the to-be-identified large model from identifying the test query and performing targeted avoidance.

[0161] The response result obtained by the response unit 803 is an answer output by the to-be-identified large model in response to the test query in real time.

[0162] After obtaining the response result output by the to-be-identified large model for each test query by the response unit 803, the detection unit 804 obtains an anomaly score of the to-be-identified large model according to the plurality of response results.

[0163] The detection unit 804 can obtain a detection score of the to-be-identified large model corresponding to a preset dimension according to the plurality of response results, and then obtain the anomaly score of the to-be-identified large model according to the obtained detection score corresponding to the preset dimension; the preset dimension in this embodiment can be at least one of a time dimension, a space dimension, and a behavior dimension.

[0164] When obtaining the anomaly score of the to-be-identified large model according to the plurality of response results, the detection unit 804 can first obtain a first detection score according to the response time corresponding to each of the plurality of response results, and then obtain the anomaly score of the to-be-identified large model according to the first detection score; the response time is the time for the to-be-identified large model to return the response result corresponding to the test query; that is, this embodiment can detect the to-be-identified large model in the time dimension, so as to obtain the anomaly score according to the first detection score corresponding to the time dimension.

[0165] When obtaining the anomaly score of the to-be-identified large model according to the plurality of response results, the detection unit 804 can first obtain the standard answer corresponding to each of the plurality of test queries, then obtain a first word embedding vector corresponding to the response result and a second word embedding vector corresponding to the standard answer, obtain a second detection score according to the first word embedding vector and the second word embedding vector corresponding to the same test query, and finally obtain the anomaly score of the to-be-identified large model according to the second detection score; that is, this embodiment can detect the to-be-identified large model in the space dimension (i.e., the feature space), so as to obtain the anomaly score according to the second detection score corresponding to the space dimension.

[0166] The detection unit 804 can also obtain response mode change information (for example, whether the style of the response result changes) of the large model to be identified according to the multiple response results when obtaining the abnormal score of the large model to be identified according to the multiple response results, obtain query mode change information (for example, whether multiple queries are continuously input to the large model to be identified at the same time) of the large model to be identified according to the test query corresponding to the response result, then obtain a third detection score according to at least one of the obtained response mode change information and the query mode change information, and finally obtain the abnormal score of the large model to be identified according to the third detection score. That is, the embodiment can detect the large model to be identified in the behavior dimension (that is, the query behavior and the response behavior), so as to obtain the abnormal score according to the third detection score of the corresponding behavior dimension.

[0167] It can be understood that the detection unit 804 can also obtain the abnormal score of the large model to be identified according to the detection score corresponding to different detection dimensions and the weight coefficient.

[0168] The detection unit 804 can also adopt the following implementation manner when obtaining the abnormal score of the large model to be identified according to the multiple response results: obtaining multiple standard answers according to the test queries corresponding to the multiple response results respectively, obtaining the query accuracy rates corresponding to different event types according to the response results and the standard answers of the test queries corresponding to the different event types, obtaining the compliance degree score of the large model to be identified according to the query accuracy rates corresponding to the different event types and the weight coefficient, and obtaining a preset score as the abnormal score of the large model to be identified in response to determining that the compliance degree score meets the third preset requirement.

[0169] That is, the detection unit 804 obtains the compliance degree score of the large model to be identified according to the query accuracy rates corresponding to different event types and the weight coefficient, and then obtains the abnormal score of the large model to be identified according to the compliance degree score, so that in the process of obtaining the abnormal score of the large model to be identified, the event type corresponding to the test query is also combined, which can further improve the accuracy of the obtained abnormal score.

[0170] The event type in the embodiment is the type of the target event used to obtain the test query, for example, a news event type, a science and technology development event type, a culture and sports event type, and the like.

[0171] The detection unit 804 can obtain the query accuracy rate corresponding to each event type according to the response result and the standard answer of the test query corresponding to the event type when obtaining the query accuracy rate corresponding to different event types, for example, taking the ratio between the number of response results similar to the standard answer and the total number of response results as the query accuracy rate.

[0172] After obtaining the anomaly score of the to-be-identified large model by the detection unit 804, the identification unit 805 obtains the identification result that the actual version of the to-be-identified large model is not the target version in response to determining that the anomaly score meets the first preset requirement.

[0173] The identification unit 805 can determine that the anomaly score meets the first preset requirement when the anomaly score is the preset score value (i.e., the anomaly score is 1), and thus obtain the identification result that the actual version of the to-be-identified large model is not the target version.

[0174] The identification unit 805 can first obtain the anomaly level of the to-be-identified large model according to the anomaly score, and then determine that the anomaly score meets the first preset requirement in response to determining that the anomaly level is the preset level, and thus obtain the identification result that the actual version of the to-be-identified large model is not the target version.

[0175] After obtaining the identification result that the actual version of the to-be-identified large model is not the target version, the identification unit 805 can further trigger an alarm mechanism to generate alarm information; the alarm information generated in this embodiment can be a report corresponding to the identification result, or can include a processing suggestion corresponding to the identification result, etc.

[0176] After generating the alarm information, the identification unit 805 can send the generated alarm information to the corresponding personnel in real time through a communication mode such as an email or a short message.

[0177] That is, the identification unit 805 can determine that the to-be-identified large model has the phenomenon of “model traffic cheating” when it is determined that the anomaly score meets the first preset requirement, and more specifically, the supplier of the to-be-identified large model has mixed a low-version large model in the provided service, so that the actual version of the to-be-identified large model includes other versions in addition to the target version.

[0178] In the technical solutions of the present disclosure, the acquisition, storage, and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0179] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program product.

[0180] As Figure 9The diagram shown is a block diagram of an electronic device for a large-scale version of the identification method according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0181] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0182] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of displays, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0183] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the large-model version of the recognition method. For example, in some embodiments, the large-model version of the recognition method may be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as storage unit 908.

[0184] In some embodiments, portions or all of the computer program can be loaded onto the apparatus 900 via the ROM 902 and / or the communications unit 909. When the computer program is loaded onto the RAM 903 and executed by the computer unit 901, one or more steps of the method of identifying a large model version described above can be performed. Alternatively, in other embodiments, the computer unit 901 can be configured, by any other suitable means (for example, by means of firmware), to perform the method of identifying a large model version.

[0185] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0186] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general or special purpose computer, special purpose computer, or other programmable processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / operations specified in the flowchart and / or block diagram. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0187] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0189] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0190] The computer system can include clients and servers. This relationship can be. The servers are generally remote from the users and can be accessed via the Internet using a communication network. The relationship can be a client-server relationship over a communications network, and as such, the servers can be accessed by the clients using computer programs. The servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are mainframe products in the cloud computing service system, and solve the defects of large management difficulty and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server, or simply VPS). The servers can also be servers of a distributed system, or servers combined with a blockchain.

[0191] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

[0192] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for recognizing a large model version, comprising: The target time window is obtained based on the first release time of the target version of the large model to be identified; Based on the target time window, obtain multiple test queries; The multiple test queries are inserted into the business data stream of the large model to be identified in real time to obtain the response results output by the large model to be identified for each test query; Based on multiple response results, the anomaly score of the large model to be identified is obtained; In response to determining that the abnormal score meets the first preset requirement, the identification result is obtained that the actual version of the large model to be identified is not the target version.

2. The method according to claim 1, wherein, The process of obtaining the target time window based on the first release time of the target version of the large model to be identified includes: Based on the target version of the large model to be identified, determine the historical version of the large model to be identified, and obtain the second release time of the historical version; Based on the first publication time and the second publication time, determine the first knowledge cutoff time and the second knowledge cutoff time of the large model to be identified; The target time window is obtained based on the first knowledge deadline and the second knowledge deadline.

3. The method according to claim 2, wherein, The step of obtaining the target time window based on the first knowledge deadline and the second knowledge deadline includes: Based on the model type of the large model to be identified, obtain the time buffer parameter; The target time window is obtained based on the first knowledge deadline, the second knowledge deadline, and the time buffer parameter.

4. The method according to claim 1, wherein, The step of obtaining multiple test queries based on the target time window includes: Multiple events whose occurrence time falls within the target time window are selected as multiple candidate events; Each of the multiple candidate events is evaluated for quality, and a quality score is obtained for each candidate event. Multiple candidate events whose quality scores meet the second preset requirements are selected as multiple target events; Based on the multiple target events, the multiple test queries are obtained.

5. The method according to claim 4, wherein, The process of evaluating the quality of the multiple candidate events to obtain a quality score for each candidate event includes: For each candidate event, obtain the first score, second score, and third score corresponding to that candidate event; The importance score of the candidate event is obtained based on the first score, the second score, and the third score. Based on the importance score, the quality score of the candidate event is obtained.

6. The method according to claim 4, wherein, The process of obtaining the multiple test queries based on the multiple target events includes: For each target event, generate a question corresponding to that target event based on its content; Use the generated question as a test query for the target event.

7. The method according to claim 4, further comprising: For each test query, the test query is input into multiple candidate large models, and multiple candidate answers are obtained based on the output results of the multiple candidate large models; The candidate answers are clustered according to their semantics to obtain at least one answer set; Select one candidate answer from the set of answers containing the most candidate answers, and use it as the standard answer for the test query.

8. The method according to claim 1, wherein, The step of inserting the multiple test queries into the business data stream of the large model to be identified in real time includes: Obtain the historical query request frequency of the large model to be identified; Based on the historical query request frequency, the predicted query request frequency for the corresponding target time is obtained; Based on the predicted query request frequency and the preset insertion weight value, the target insertion frequency corresponding to the target time is obtained; According to the target insertion frequency, the multiple test queries are inserted into the business data stream of the large model to be identified in real time at the target time.

9. The method according to claim 1, further comprising: The multiple test queries are disguised to obtain the query disguise results corresponding to the multiple test queries respectively; Multiple query results are disguised and inserted into the business data stream of the large model to be identified in real time.

10. The method according to claim 1, wherein, The process of obtaining the anomaly score of the large model to be identified based on multiple response results includes: Based on the multiple response results, obtain the detection score of the large model to be identified corresponding to the preset dimension; Based on the detection scores, the anomaly score of the large model to be identified is obtained.

11. The method according to claim 1, wherein, The process of obtaining the anomaly score of the large model to be identified based on multiple response results includes: Based on the test queries corresponding to the multiple response results, multiple standard answers are obtained; Based on the response results of test queries for different event types and the standard answers, the query accuracy for each event type is obtained. Based on the query accuracy and weight coefficients corresponding to different event types, the compliance score of the large model to be identified is obtained; In response to determining that the compliance score meets the third preset requirement, a preset score is obtained as the anomaly score of the large model to be identified.

12. The method according to claim 1, wherein, Determining that the abnormal score meets the first preset requirement includes: Based on the anomaly score, the anomaly level of the large model to be identified is obtained; In response to determining that the anomaly level is a preset level, the anomaly score is determined to meet the first preset requirement.

13. A large-scale model version recognition device, comprising: The processing unit is used to obtain the target time window based on the first release time of the target version of the large model to be identified; The acquisition unit is used to acquire multiple test queries based on the target time window; The response unit is used to insert the multiple test queries into the business data stream of the large model to be identified in real time, and obtain the response result output by the large model to be identified for each test query. The detection unit is used to obtain the anomaly score of the large model to be identified based on multiple response results; The identification unit is configured to, in response to determining that the abnormal score meets a first preset requirement, obtain an identification result that the actual version of the large model to be identified is not the target version.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-12.

Images