An intelligence information utility evaluation method and system based on artificial intelligence
By obtaining relevant data and historical intelligence information of intelligence information, using artificial intelligence models to evaluate the timeliness and accuracy of intelligence information, the problem of insufficient comprehensive intelligence information evaluation in the existing technology is solved, and efficient and accurate evaluation of intelligence information is achieved.
Patent Information
- Application Number
- CN202411211665.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-30
AI Technical Summary
The prior art cannot conduct a comprehensive assessment based on the timeliness and accuracy of intelligence information, resulting in inaccurate information utility evaluation.
By obtaining relevant data of intelligence information, determine its timeliness and accuracy scores, including intelligence information release time, update speed, update frequency, key entities and information sources, use artificial intelligence models for evaluation, and combine historical intelligence information training accuracy evaluation The model, calculates the timeliness and accuracy scores, and finally determines the utility score of intelligence information.
It improves the objectivity and comprehensiveness of the timeliness and accuracy assessment of intelligence information, ensures the accuracy and comprehensiveness of the utility assessment of intelligence information, and enhances the objectivity and accuracy of information assessment.
Smart Images

Figure CN118710090B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligence information utility evaluation method and system based on artificial intelligence. Background Art
[0002] In the related art, CN116029576A discloses a method and system for evaluating the decision-making utility of earthquake emergency information. By combining the user's preference for emergency information, the amount of information contained in the emergency information itself, and the expert's rating of the emergency information, an evaluation model that can evaluate the decision-making utility of emergency information is fitted. The evaluation model can evaluate the existing emergency information set and then delete the low-utility emergency information. It can also evaluate the utility of the latest emergency information at any time based on the acquired user preference and information volume data, so as to retain the high-utility emergency information to enrich the emergency information set, thereby providing users with high-quality emergency information after the earthquake. Through this evaluation model, the emergency information is numerically quantified, and the user can intuitively obtain the utility of the emergency information. The scheme also updates the evaluation model according to the actual utility value of the emergency information fed back by the user after the earthquake, so that the emergency information obtained based on the evaluation model is more in line with the user's preference and the service efficiency is higher.
[0003] CN104392332A discloses a method for online evaluation of technical information, including: obtaining the technical information to be evaluated; judging whether the technical information conforms to a preset evaluation formula and evaluation parameters; if the technical information conforms to the preset evaluation formula and evaluation parameters, outputting the technical information; if the technical information does not conform to the preset evaluation formula and evaluation parameters, feeding back the technical information that does not conform to the preset evaluation formula and evaluation parameters. Online evaluation is to obtain the technical information to be evaluated through the network, without printing and submitting paper documents, saving the time cost of evaluation, printing cost and transportation cost; judging whether the technical information conforms to the preset evaluation formula and evaluation parameters, automatic detection and other means can be used for evaluation, and the operator only needs to pre-set the evaluation formula and evaluation parameters corresponding to the technical information to be evaluated, avoiding errors and omissions caused by manual evaluation, and improving the accuracy of technical information evaluation.
[0004] Based on the above related technologies, the effectiveness of information can be evaluated through evaluation models. However, the related technologies do not consider the importance of the timeliness of information in the evaluation of information utility, that is, it is impossible to comprehensively evaluate the utility of intelligence information based on the timeliness and accuracy of intelligence information.
[0005] The information disclosed in the background technology section of this application is only intended to deepen the understanding of the general background technology of this application, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the invention
[0006] The present invention provides an intelligence information utility evaluation method and system based on artificial intelligence, which can solve the technical problem that related technologies cannot comprehensively evaluate the utility of intelligence information based on the timeliness and accuracy of intelligence information.
[0007] According to a first aspect of the present invention, there is provided an intelligence information utility evaluation method based on artificial intelligence, comprising:
[0008] Acquire relevant data of intelligence information, wherein the relevant data includes: intelligence information release time, intelligence information update speed, intelligence information update frequency and intelligence information source information;
[0009] Determine key entities of the event of the intelligence information according to the intelligence information, wherein the key entities include: the time when the intelligence event occurred, the location where the intelligence event occurred, and persons related to the intelligence event;
[0010] Based on the intelligence information, identify similar intelligence information from multiple sources of intelligence information;
[0011] Acquire similar key entities of the event of the similar intelligence information, wherein the similar key entities include: the time when the similar intelligence event occurred, the location where the similar intelligence event occurred, and persons related to the similar intelligence event, and determine the similar intelligence information source information of the similar intelligence information;
[0012] Determine the timeliness score of the intelligence information according to the release time of the intelligence information, the update speed of the intelligence information, and the update frequency of the intelligence information;
[0013] Acquire multiple historical intelligence information, and train an intelligence information accuracy assessment model based on the historical intelligence information to obtain a trained intelligence information accuracy assessment model;
[0014] Processing the similar intelligence information source information, the intelligence information source information, the key entity and the similar key entity through a trained intelligence information accuracy assessment model to determine an accuracy score of the intelligence information;
[0015] A usefulness score of the intelligence information is determined based on the timeliness score and the accuracy score.
[0016] According to a second aspect of the present invention, there is provided an intelligence information utility evaluation system based on artificial intelligence, comprising:
[0017] A relevant data acquisition module is used to acquire relevant data of intelligence information, wherein the relevant data includes: intelligence information release time, intelligence information update speed, intelligence information update frequency and intelligence information source information;
[0018] A key entity module is used to determine the key entities of the intelligence information event based on the intelligence information, wherein the key entities include: the time when the intelligence event occurred, the location where the intelligence event occurred, and the persons related to the intelligence event;
[0019] A similar intelligence information module is used to determine similar intelligence information of multiple information sources of the intelligence information based on the intelligence information;
[0020] A similar key entity module is used to obtain similar key entities of the event of the similar intelligence information, wherein the similar key entities include: the time when the similar intelligence event occurred, the location where the similar intelligence event occurred and the people related to the similar intelligence event, and determine the similar intelligence information source information of the similar intelligence information;
[0021] A timeliness scoring module, used to determine the timeliness score of the intelligence information according to the release time of the intelligence information, the update speed of the intelligence information, and the update frequency of the intelligence information;
[0022] A training model module is used to obtain multiple historical intelligence information and train an intelligence information accuracy assessment model based on the historical intelligence information to obtain a trained intelligence information accuracy assessment model;
[0023] An accuracy scoring module, used to process the similar intelligence information source information, the intelligence information source information, the key entity and the similar key entity through a trained intelligence information accuracy assessment model to determine an accuracy score of the intelligence information;
[0024] The utility scoring module is used to determine the utility score of the intelligence information based on the timeliness score and the accuracy score.
[0025] Technical effect: According to the present invention, the timeliness of intelligence information can be accurately analyzed, the intelligence information accuracy assessment model can be trained according to historical intelligence information, and the accuracy of intelligence information can be assessed based on the trained intelligence information accuracy assessment model, thereby improving the objectivity and accuracy of the intelligence information accuracy assessment. Furthermore, the usefulness score of intelligence information can be determined according to the timeliness and accuracy of intelligence information, thereby improving the comprehensiveness and accuracy of the usefulness assessment of intelligence information. When determining the timeliness score of intelligence information, the timeliness score of intelligence information can be determined according to the release time of intelligence information, the update speed of intelligence information, and the update frequency of intelligence information. In the calculation process, the influence of the three aspects of the time lag of intelligence information, the update speed of intelligence information, and the update frequency of intelligence information on the timeliness of intelligence information is fully considered, thereby improving the comprehensiveness and accuracy of the timeliness score. When determining the credibility score of the source of historical intelligence information, the credibility score of the source of historical intelligence information can be determined based on the information of the publisher of historical intelligence information, the identification results of the news release agency, the media identification results and the identification results of the academic institution. In the calculation process, the credibility of the source of historical intelligence information can be evaluated from two aspects: the credibility of the publisher of historical intelligence information and the credibility of the agency that releases historical intelligence information, which improves the comprehensiveness and accuracy of the credibility score of the source of historical intelligence information and provides a data basis for the training of the intelligence information accuracy evaluation model. When determining the training loss function of the intelligence information accuracy evaluation model, the influence of the above data on the error of the accuracy score of the sample historical intelligence information can be determined by the similarity between the historical intelligence information and the historical similar intelligence information and the influence of the source credibility score of the historical intelligence information on the accuracy of the historical intelligence information. Based on this influence and the relative error of the accuracy score of the sample historical intelligence information, the training loss function is set to reduce the training loss function of the intelligence information accuracy evaluation model during the training process, thereby improving the accuracy of the intelligence information accuracy evaluation model.
[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present invention. Other features and aspects of the present invention will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other embodiments can be obtained based on these drawings without creative work.
[0028] Figure 1A flowchart of an intelligence information utility evaluation method based on artificial intelligence according to an embodiment of the present invention is exemplarily shown;
[0029] Figure 2 A block diagram of an intelligence information utility evaluation system based on artificial intelligence according to an embodiment of the present invention is exemplarily shown. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0031] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0032] Figure 1 A flowchart of an artificial intelligence-based intelligence information utility evaluation method according to an embodiment of the present invention is exemplarily shown, wherein the method comprises:
[0033] Step S101, obtaining relevant data of intelligence information, wherein the relevant data includes: intelligence information release time, intelligence information update speed, intelligence information update frequency and intelligence information source information;
[0034] Step S102, determining key entities of the event of the intelligence information according to the intelligence information, wherein the key entities include: the time when the intelligence event occurred, the location where the intelligence event occurred, and persons related to the intelligence event;
[0035] Step S103, determining similar intelligence information from multiple information sources of the intelligence information based on the intelligence information;
[0036] Step S104, obtaining similar key entities of the event of the similar intelligence information, wherein the similar key entities include: the time when the similar intelligence event occurred, the location where the similar intelligence event occurred and the persons related to the similar intelligence event, and determining the similar intelligence information source information of the similar intelligence information;
[0037] Step S105, determining the timeliness score of the intelligence information according to the release time of the intelligence information, the update speed of the intelligence information, and the update frequency of the intelligence information;
[0038] Step S106, obtaining a plurality of historical intelligence information, and training an intelligence information accuracy assessment model according to the historical intelligence information to obtain a trained intelligence information accuracy assessment model;
[0039] Step S107, processing the similar intelligence information source information, the intelligence information source information, the key entity and the similar key entity through the trained intelligence information accuracy assessment model to determine the accuracy score of the intelligence information;
[0040] Step S108, determining the usefulness score of the intelligence information according to the timeliness score and the accuracy score.
[0041] According to the artificial intelligence-based intelligence information utility evaluation method of an embodiment of the present invention, the timeliness of intelligence information can be accurately analyzed, an intelligence information accuracy evaluation model can be trained according to historical intelligence information, and the accuracy of intelligence information can be evaluated based on the trained intelligence information accuracy evaluation model, thereby improving the objectivity and accuracy of intelligence information accuracy evaluation. Furthermore, based on the timeliness and accuracy of the intelligence information, the utility score of the intelligence information is determined, thereby improving the comprehensiveness and accuracy of the intelligence information utility evaluation.
[0042] According to an embodiment of the present invention, in step S101, relevant data of intelligence information is obtained, wherein the relevant data includes: intelligence information release time, intelligence information update speed, intelligence information update frequency and intelligence information source information.
[0043] For example, obtain the release time, update speed, update frequency and source information of intelligence information. The update speed of intelligence information refers to the time required for intelligence information to be generated, acquired, processed and delivered to end users by intelligence agencies. The update speed of intelligence information directly affects whether users can obtain the latest information in a timely manner. The update frequency of intelligence information refers to the number of times intelligence information is updated or replaced within a period of time.
[0044] According to an embodiment of the present invention, in step S102, key entities of an event of the intelligence information are determined based on the intelligence information, wherein the key entities include: the time when the intelligence event occurred, the location where the intelligence event occurred, and persons related to the intelligence event.
[0045] For example, through the entity recognition module in natural language processing technology, intelligence information can be automatically processed to quickly identify key entities in the intelligence information, such as time, place, and names.
[0046] According to an embodiment of the present invention, in step S103, similar intelligence information from multiple information sources of the intelligence information is determined based on the intelligence information.
[0047] For example, keyword searches are performed on other information sources (e.g., news websites, social media, and forums) based on the time, place, and names of people in the identified intelligence information to identify similar intelligence information.
[0048] According to one embodiment of the present invention, in step S104, similar key entities of the event of the similar intelligence information are obtained, wherein the similar key entities include: the time when the similar intelligence event occurred, the place where the similar intelligence event occurred and the persons related to the similar intelligence event, and the similar intelligence information source information of the similar intelligence information is determined.
[0049] For example, through the entity recognition module in natural language processing technology, similar intelligence information can be automatically processed and entities in similar intelligence information can be quickly identified.
[0050] According to an embodiment of the present invention, in step S105, the timeliness score of the intelligence information is determined according to the release time of the intelligence information, the update speed of the intelligence information, and the update frequency of the intelligence information.
[0051] For example, based on the release time, update speed, and update frequency of intelligence information, the timeliness of intelligence information is evaluated to determine the timeliness score of the intelligence information.
[0052] According to an embodiment of the present invention, step S105 includes: determining the timeliness score of the intelligence information according to formula (1): ,
[0053] (1)
[0054] in, , and is the preset weight, The time when the intelligence event occurred. The time for intelligence information release. For the speed of intelligence information update, The frequency of intelligence information updates, is the preset hysteresis threshold, To preset the intelligence information update speed threshold, Set the frequency threshold for updating intelligence information.
[0055] According to one embodiment of the present invention, is the difference between the time when intelligence information is released and the time when the intelligence event occurs, indicating the time lag of intelligence information. is the relative difference between the preset time lag threshold and the time lag of intelligence information. The larger the ratio is, the shorter the time lag of intelligence information is and the stronger the timeliness of intelligence information is. It is the ratio of the intelligence information update speed to the preset intelligence information update speed threshold. The larger the ratio is, the faster the intelligence information is updated, the earlier the user can grasp the situation, and the more timely the intelligence information is. It is the ratio of the intelligence information update frequency to the preset intelligence information update frequency threshold. The larger the ratio is, the higher the intelligence information update frequency is, the more timely users can obtain the latest intelligence information, and the more timely the intelligence information is. The timeliness score of intelligence information is determined by taking a weighted sum based on the time lag, update speed and update frequency of intelligence information.
[0056] In this way, the timeliness score of intelligence information can be determined based on the release time, update speed and update frequency of intelligence information. During the calculation process, the impact of the three aspects of intelligence information time lag, intelligence information update speed and intelligence information update frequency on the timeliness of intelligence information is fully considered, thereby improving the comprehensiveness and accuracy of the timeliness score.
[0057] According to an embodiment of the present invention, in step S106, a plurality of historical intelligence information is obtained, and an intelligence information accuracy assessment model is trained based on the historical intelligence information to obtain a trained intelligence information accuracy assessment model.
[0058] For example, the intelligence information accuracy assessment model is a type of deep learning neural network model. The intelligence information accuracy assessment model can be trained through historical intelligence information and related data to obtain a trained intelligence information accuracy assessment model. The trained intelligence information accuracy assessment model can more accurately assess the accuracy of intelligence information.
[0059] According to one embodiment of the present invention, step S106 includes:
[0060] Acquire historical intelligence information source information of multiple historical intelligence information;
[0061] Determining a credibility score of the historical intelligence information source based on the historical intelligence information source information;
[0062] The intelligence information accuracy assessment model is trained according to the historical intelligence information and the credibility score of the source of the historical intelligence information to obtain a trained intelligence information accuracy assessment model.
[0063] For example, determine the historical intelligence information source information of the historical intelligence information, and based on the historical intelligence information source information, evaluate the credibility of the historical intelligence information and determine the historical intelligence information source credibility score; train the intelligence information accuracy assessment model based on the historical intelligence information and the historical intelligence information source credibility score to obtain a trained intelligence information accuracy assessment model.
[0064] According to an embodiment of the present invention, determining the credibility score of the source of historical intelligence information according to the source information of the historical intelligence information includes:
[0065] Determine the publisher information of the historical intelligence information according to the source information of the historical intelligence information, wherein the publisher information of the historical intelligence information includes: the accuracy score of the publisher's historical published information and the number of times the publisher has published historical information;
[0066] Determine the name of the source of the historical intelligence information according to the source information of the historical intelligence information;
[0067] Obtain the names of known news release agencies, known media, and known academic institutions;
[0068] Determine the identification result of the message issuing agency according to the name of the source of the historical intelligence information and the name of the known message issuing agency;
[0069] Determining a media identification result according to the historical intelligence information source name and the known media name;
[0070] Determine an academic institution identification result according to the name of the historical intelligence information source and the name of the known academic institution;
[0071] The credibility score of the source of historical intelligence information is determined based on the publisher information of the historical intelligence information, the identification result of the message publishing organization, the media identification result and the identification result of the academic institution.
[0072] For example, based on the source information of historical intelligence information, the accuracy score of the publisher's historical information releases and the number of times the publisher has released information are determined. The accuracy score of the publisher's historical information releases is the accuracy score rated by experts in the relevant field for the historical intelligence information released by the publisher, and the number of times the publisher has released information is the number of times the publisher has released intelligence information, which can reflect the publisher's work experience. Based on the comparison between the name of the source of historical intelligence information and the names of multiple known news release agencies, if the name of the source of historical intelligence information belongs to the name of a known news release agency, the news release agency identification result is 1, indicating that the source of historical intelligence information has a high degree of credibility. Otherwise, the news release agency identification result is 0. Based on the historical intelligence information The source name is compared with multiple known media names. If the source name of the historical intelligence information belongs to a known media name, the media identification result is 1, indicating that the source of the historical intelligence information has a high credibility. Otherwise, the media identification result is 0. The source name of the historical intelligence information is compared with multiple known academic institution names. If the source name of the historical intelligence information belongs to a known academic institution name, the academic institution identification result is 1, indicating that the source of the historical intelligence information has a high credibility. Otherwise, the academic institution identification result is 0. Based on the information of the publisher of the historical intelligence information, the identification result of the message publishing agency, the media identification result and the academic institution identification result, the credibility of the source of the historical intelligence information is evaluated to determine the credibility score of the source of the historical intelligence information.
[0073] According to an embodiment of the present invention, the credibility score of the source of historical intelligence information is determined based on the information of the publisher of the historical intelligence information, the identification result of the message publishing organization, the media identification result and the identification result of the academic institution, including: determining the credibility score of the source of historical intelligence information of the i-th historical intelligence information according to formula (2): ,
[0074] (2)
[0075] in, and is the preset weight, is the identification result of the news release agency of the i-th historical intelligence information, is the media recognition result of the i-th historical intelligence information, is the academic institution identification result of the i-th historical intelligence information, is the preset release count threshold, Score the accuracy of historical information released by the publisher of the i-th historical intelligence information for the publisher of the k-th information released, To preset the accuracy score threshold, , , , K is the number of times the publisher has published information in history, k≤K, k and K are both positive integers.
[0076] According to one embodiment of the present invention, , It is the ratio of the publisher's historical information release times to the preset release times threshold. The larger the ratio is, the more experience the publisher of the historical intelligence information has, and the higher the credibility of the source of the historical intelligence information. is the average accuracy score of the publisher's historical information published by the publisher of the i-th historical intelligence information for the K times, It is the ratio of the average accuracy score of the publisher's historical information published by the publisher of the i-th historical intelligence information for the K times and the preset accuracy score threshold. The larger the ratio is, the higher the accuracy of the intelligence information published by the publisher of the historical intelligence information is, and the higher the credibility of the source of the historical intelligence information is. Indicates the credibility of the publisher. The larger the ratio, the higher the credibility of the intelligence information released by the publisher. is the sum of the identification results of the news release agency, media identification results, and academic institution identification results of the i-th historical intelligence information, indicating the credibility of the release agency of the i-th historical intelligence information. It means that the credibility of the source of the historical intelligence information is determined according to the credibility of the publisher of the i-th historical intelligence information and the credibility of the issuing organization.
[0077] In this way, the credibility score of the source of historical intelligence information can be determined based on the information of the publisher of historical intelligence information, the identification results of the news publishing agency, the media identification results and the identification results of the academic institution. During the calculation process, the credibility of the source of historical intelligence information can be evaluated from two aspects: the credibility of the publisher of historical intelligence information and the credibility of the publishing agency of historical intelligence information. This improves the comprehensiveness and accuracy of the credibility score of the source of historical intelligence information and provides a data basis for the training of the intelligence information accuracy evaluation model.
[0078] According to an embodiment of the present invention, training an intelligence information accuracy assessment model based on the historical intelligence information and the credibility score of the source of the historical intelligence information to obtain a trained intelligence information accuracy assessment model includes:
[0079] Determine historical key entities of events in the historical intelligence information according to the historical intelligence information, wherein the historical key entities include: the time when the historical intelligence event occurred, the location where the historical intelligence event occurred, and persons related to the historical intelligence event;
[0080] Determining historical similar intelligence information from multiple information sources of the historical intelligence information based on the historical intelligence information;
[0081] Acquire historical similar key entities of the event of the historical similar intelligence information, wherein the historical similar key entities include: the time when the historical similar intelligence event occurred, the location where the historical similar intelligence event occurred, and people related to the historical similar intelligence event;
[0082] Determine the source information of historical similar intelligence information;
[0083] Determine the credibility score of the historical similar intelligence information source based on the historical similar intelligence information source information;
[0084] The time when the historical intelligence event occurred, the location where the historical intelligence event occurred, and the people related to the historical intelligence event are processed by a natural language processing model to determine a historical intelligence event vector;
[0085] The time when the historical similar intelligence event occurred, the location where the historical similar intelligence event occurred, and the people related to the historical similar intelligence event are processed by a natural language processing model to determine a historical similar intelligence event vector;
[0086] Through the intelligence information accuracy evaluation model, the historical intelligence information source credibility scores, historical similar intelligence information source information, historical intelligence event vectors and historical similar intelligence event vectors of multiple historical intelligence information are processed to determine the sample historical intelligence information accuracy scores of multiple historical intelligence information;
[0087] Determining expert historical intelligence information accuracy scores for multiple historical intelligence information;
[0088] Determine a training loss function of an intelligence information accuracy assessment model based on the sample historical intelligence information accuracy score, the expert historical intelligence information accuracy score, the historical intelligence information source credibility score, the historical similar intelligence information source credibility score, the historical intelligence event vector, and the historical similar intelligence event vector;
[0089] The intelligence information accuracy assessment model is trained according to the training loss function to obtain a trained intelligence information accuracy assessment model.
[0090] For example, the historical intelligence information is processed through the entity recognition module in the natural language processing technology to determine the time of occurrence of the historical intelligence event, the location of the historical intelligence event and the people related to the historical intelligence event; in other multiple information sources, a keyword search is performed based on the time of occurrence of the historical intelligence event, the location of the historical intelligence event and the people related to the historical intelligence event to determine the historical similar intelligence information, and determine the time of occurrence of the historical similar intelligence event, the location of occurrence of the historical similar intelligence event and the people related to the historical similar intelligence event, as well as the historical similar intelligence information source information of the historical similar intelligence information; based on the historical similar intelligence information source information, determine the credibility score of the historical similar intelligence information source, and its calculation method is similar to the credibility score of the historical intelligence information source in formula (2), which will not be repeated here; through the natural language processing model, the time of occurrence of the historical intelligence event, the location of the historical intelligence event, the people related to the historical intelligence event, the time of occurrence of the historical similar intelligence event, the location of the historical similar intelligence event and the people related to the historical similar intelligence event are converted into vector form to determine the historical intelligence event vector and the historical similar intelligence event vector; the intelligence information accuracy evaluation model is a deep learning neural network model, which can be used to evaluate the intelligence information accuracy. The credibility scores of historical intelligence information sources, historical similar intelligence information source information, historical intelligence event vectors and historical similar intelligence event vectors of multiple historical intelligence information are processed to determine the accuracy scores of sample historical intelligence information of multiple historical intelligence information; the accuracy scores of expert historical intelligence information evaluated by experts in related fields for multiple historical intelligence information are obtained; the experts in related fields use a combination of qualitative (evaluating the accuracy of intelligence information through subjective judgment of experts) and quantitative (e.g., bibliometrics and data envelopment analysis, evaluating the accuracy of intelligence information through quantitative analysis of intelligence information) methods to evaluate the accuracy of intelligence information; the experts evaluate the content of intelligence information based on the actual needs of intelligence work to obtain the qualitative accuracy score of intelligence information; the experts use bibliometrics to quantitatively analyze the frequency of citations of intelligence sources, the number of times intelligence content is cited by other authoritative documents and the reputation of intelligence publishing institutions to obtain the quantitative accuracy score of intelligence information; the weighted summation of the qualitative accuracy score of intelligence information and the quantitative accuracy score of intelligence information is performed to determine the accuracy score of single expert historical intelligence information; multiple experts are independently evaluated in an anonymous manner; the single expert historical intelligence information accuracy scores of multiple experts are collected and averaged to form the final expert historical intelligence information accuracy score;According to the accuracy score of sample historical intelligence information, the accuracy score of expert historical intelligence information, the credibility score of historical intelligence information source, the credibility score of historical similar intelligence information source, the historical intelligence event vector and the historical similar intelligence event vector, the training loss function of the intelligence information accuracy evaluation model is determined, and the intelligence information accuracy evaluation model is trained through the training loss function to obtain the trained load forecasting model. ;
[0091] According to one embodiment of the present invention, the training loss function of the intelligence information accuracy evaluation model is determined based on the accuracy score of the sample historical intelligence information, the accuracy score of the expert historical intelligence information, the credibility score of the historical similar intelligence information source, the historical intelligence event vector and the historical similar intelligence event vector, including: determining the training loss function of the intelligence information accuracy evaluation model according to formula (3): ,
[0092] (3)
[0093] in, Score the accuracy of the expert historical intelligence information for the i-th historical intelligence information, Score the accuracy of the sample historical intelligence information for the i-th historical intelligence information, is the time when the historical intelligence event of the i-th historical intelligence information occurred, is the location of the historical intelligence event of the i-th historical intelligence information, is the person related to the historical intelligence event of the i-th historical intelligence information, is the historical intelligence event vector of the i-th historical intelligence information, for The transposed vector of is the time of occurrence of the historical similar intelligence event of the historical similar intelligence information of the jth information source of the i-th historical intelligence information, is the location of the historical similar intelligence event of the historical similar intelligence information of the jth information source of the i-th historical intelligence information, is the person related to the historical similar intelligence event of the event of the historical similar intelligence information of the jth information source of the i-th historical intelligence information, Score the credibility of the historical similar intelligence information source of the j-th information source of the i-th historical intelligence information, is the historical similar intelligence event vector of the historical similar intelligence information of the jth information source of the i-th historical intelligence information, is the credibility score of the historical intelligence information source of the i-th historical intelligence information, m is the number of information sources, n is the number of historical intelligence information, i≤n, j≤m, i, j, m and n are positive integers.
[0094] According to one embodiment of the present invention, is the cosine similarity between the historical intelligence event vector of the ith historical intelligence information and the historical similar intelligence event vector of the historical similar intelligence information of the jth information source of the ith historical intelligence information. The larger the cosine similarity, the smaller the difference between the time, place and person information in the ith historical intelligence information and the time, place and person information in the historical similar intelligence information of the jth information source of the ith historical intelligence information. is the product of the above cosine similarity and the historical similar intelligence information source credibility score of the historical similar intelligence information of the j-th information source of the ith historical intelligence information, indicating the content accuracy of the ith historical intelligence information determined based on the historical similar intelligence information source credibility of the historical similar intelligence information of the j-th information source of the ith historical intelligence information and the difference between the historical intelligence information and the historical similar intelligence information, In order to find the average value according to the number of information sources, the above-mentioned operation of finding the average value is to cross-validate the historical intelligence information and the historical similar intelligence information from multiple information sources. The higher the average value, the higher the accuracy of the content of the i-th historical intelligence information. The credibility score of the historical intelligence information source of the i-th historical intelligence information and the accuracy of the content of the i-th historical intelligence information are positively correlated with the accuracy of the i-th historical intelligence information. For example, when the credibility of the information source of the i-th historical intelligence information is higher and the accuracy of the content of the i-th historical intelligence information is higher, the accuracy of the i-th historical intelligence information is higher, that is, , The larger the value of , the higher the accuracy of the i-th historical intelligence information, and the greater the impact of the error on the accuracy score of the sample historical intelligence information. is the relative error between the expert historical intelligence information accuracy score of the i-th historical intelligence information and the sample historical intelligence information accuracy score of the i-th historical intelligence information.
[0095] According to one embodiment of the present invention, using The relative errors of the accuracy scores of the sample historical intelligence information of each historical intelligence information are weighted averaged to obtain the training loss function. During the training process, the above training loss function is reduced, so that the error between the accuracy score of the expert historical intelligence information and the accuracy score of the sample historical intelligence information is reduced, and the accuracy of the intelligence information accuracy assessment model for the accuracy assessment of intelligence information is improved, thereby improving the accuracy of the intelligence information accuracy assessment model.
[0096] In this way, the influence of the above data on the error of the accuracy score of sample historical intelligence information can be determined by the similarity between historical intelligence information and similar historical intelligence information, and the influence of the source credibility score of historical intelligence information on the accuracy of historical intelligence information. Based on this influence and the relative error of the accuracy score of sample historical intelligence information, the training loss function is set to reduce the training loss function of the intelligence information accuracy evaluation model during the training process, thereby improving the accuracy of the intelligence information accuracy evaluation model.
[0097] According to one embodiment of the present invention, in step S107, the similar intelligence information source information, the intelligence information source information, the key entity and the similar key entity are processed by a trained intelligence information accuracy assessment model to determine the accuracy score of the intelligence information.
[0098] For example, through the trained intelligence information accuracy assessment model, similar intelligence information source information, intelligence information source information, key entities and similar key entities are processed to evaluate the accuracy of the intelligence information and determine the accuracy score of the intelligence information.
[0099] According to an embodiment of the present invention, in step S108, the usefulness score of the intelligence information is determined based on the timeliness score and the accuracy score.
[0100] For example, the usefulness score of intelligence information is determined by taking a weighted sum of the timeliness score and the accuracy score of intelligence information.
[0101] According to the artificial intelligence-based intelligence information utility evaluation method of the embodiment of the present invention, the timeliness of intelligence information can be accurately analyzed, the intelligence information accuracy evaluation model is trained according to historical intelligence information, and the accuracy of intelligence information is evaluated based on the trained intelligence information accuracy evaluation model, thereby improving the objectivity and accuracy of the intelligence information accuracy evaluation. Furthermore, the utility score of intelligence information is determined according to the timeliness and accuracy of intelligence information, thereby improving the comprehensiveness and accuracy of the intelligence information utility evaluation. When determining the timeliness score of intelligence information, the timeliness score of intelligence information can be determined according to the release time of intelligence information, the update speed of intelligence information, and the update frequency of intelligence information. In the calculation process, the influence of the three aspects of the time lag of intelligence information, the update speed of intelligence information, and the update frequency of intelligence information on the timeliness of intelligence information is fully considered, thereby improving the comprehensiveness and accuracy of the timeliness score. When determining the credibility score of the source of historical intelligence information, the credibility score of the source of historical intelligence information can be determined based on the information of the publisher of historical intelligence information, the identification results of the news release agency, the media identification results and the identification results of the academic institution. In the calculation process, the credibility of the source of historical intelligence information can be evaluated from two aspects: the credibility of the publisher of historical intelligence information and the credibility of the agency that releases historical intelligence information, which improves the comprehensiveness and accuracy of the credibility score of the source of historical intelligence information and provides a data basis for the training of the intelligence information accuracy evaluation model. When determining the training loss function of the intelligence information accuracy evaluation model, the influence of the above data on the error of the accuracy score of the sample historical intelligence information can be determined by the similarity between the historical intelligence information and the historical similar intelligence information and the influence of the source credibility score of the historical intelligence information on the accuracy of the historical intelligence information. Based on this influence and the relative error of the accuracy score of the sample historical intelligence information, the training loss function is set to reduce the training loss function of the intelligence information accuracy evaluation model during the training process, thereby improving the accuracy of the intelligence information accuracy evaluation model.
[0102] Figure 2 A block diagram of an intelligence information utility evaluation system based on artificial intelligence according to an embodiment of the present invention is exemplarily shown, wherein the system comprises:
[0103] A relevant data acquisition module is used to acquire relevant data of intelligence information, wherein the relevant data includes: intelligence information release time, intelligence information update speed, intelligence information update frequency and intelligence information source information;
[0104] A key entity module is used to determine the key entities of the intelligence information event based on the intelligence information, wherein the key entities include: the time when the intelligence event occurred, the location where the intelligence event occurred, and the persons related to the intelligence event;
[0105] A similar intelligence information module is used to determine similar intelligence information of multiple information sources of the intelligence information based on the intelligence information;
[0106] A similar key entity module is used to obtain similar key entities of the event of the similar intelligence information, wherein the similar key entities include: the time when the similar intelligence event occurred, the location where the similar intelligence event occurred and the people related to the similar intelligence event, and determine the similar intelligence information source information of the similar intelligence information;
[0107] A timeliness scoring module, used to determine the timeliness score of the intelligence information according to the release time of the intelligence information, the update speed of the intelligence information, and the update frequency of the intelligence information;
[0108] A training model module is used to obtain multiple historical intelligence information and train an intelligence information accuracy assessment model based on the historical intelligence information to obtain a trained intelligence information accuracy assessment model;
[0109] An accuracy scoring module, used to process the similar intelligence information source information, the intelligence information source information, the key entity and the similar key entity through a trained intelligence information accuracy assessment model to determine an accuracy score of the intelligence information;
[0110] The utility scoring module is used to determine the utility score of the intelligence information based on the timeliness score and the accuracy score.
[0111] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.
[0112] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.
Claims
1. An intelligence information utility evaluation method based on artificial intelligence, characterized in that: include: Acquire relevant data of intelligence information, wherein the relevant data includes: intelligence information release time, intelligence information update speed, intelligence information update frequency and intelligence information source information; Determine key entities of the event of the intelligence information according to the intelligence information, wherein the key entities include: the time when the intelligence event occurred, the location where the intelligence event occurred, and persons related to the intelligence event; Based on the intelligence information, identify similar intelligence information from multiple sources of intelligence information; Acquire similar key entities of the event of the similar intelligence information, wherein the similar key entities include: the time when the similar intelligence event occurred, the location where the similar intelligence event occurred, and persons related to the similar intelligence event, and determine the similar intelligence information source information of the similar intelligence information; Determine the timeliness score of the intelligence information according to the release time of the intelligence information, the update speed of the intelligence information, and the update frequency of the intelligence information; Acquire multiple historical intelligence information, and train an intelligence information accuracy assessment model based on the historical intelligence information to obtain a trained intelligence information accuracy assessment model; Processing the similar intelligence information source information, the intelligence information source information, the key entity and the similar key entity through a trained intelligence information accuracy assessment model to determine an accuracy score of the intelligence information; Determining a usefulness score of the intelligence information based on the timeliness score and the accuracy score; Acquire multiple historical intelligence information, and train an intelligence information accuracy assessment model based on the historical intelligence information to obtain a trained intelligence information accuracy assessment model, including: Acquire historical intelligence information source information of multiple historical intelligence information; Determining a credibility score of the historical intelligence information source based on the historical intelligence information source information; Training an intelligence information accuracy assessment model according to the historical intelligence information and the credibility score of the source of the historical intelligence information to obtain a trained intelligence information accuracy assessment model; According to the historical intelligence information and the credibility score of the source of the historical intelligence information, the intelligence information accuracy assessment model is trained to obtain a trained intelligence information accuracy assessment model, including: Determine historical key entities of events in the historical intelligence information according to the historical intelligence information, wherein the historical key entities include: the time when the historical intelligence event occurred, the location where the historical intelligence event occurred, and persons related to the historical intelligence event; Determining historical similar intelligence information from multiple information sources of the historical intelligence information based on the historical intelligence information; Acquire historical similar key entities of the event of the historical similar intelligence information, wherein the historical similar key entities include: the time when the historical similar intelligence event occurred, the location where the historical similar intelligence event occurred, and people related to the historical similar intelligence event; Determine the source information of historical similar intelligence information; Determine the credibility score of the historical similar intelligence information source based on the historical similar intelligence information source information; The time when the historical intelligence event occurred, the location where the historical intelligence event occurred, and the people related to the historical intelligence event are processed by a natural language processing model to determine a historical intelligence event vector; The time when the historical similar intelligence event occurred, the location where the historical similar intelligence event occurred, and the people related to the historical similar intelligence event are processed by a natural language processing model to determine a historical similar intelligence event vector; Through the intelligence information accuracy evaluation model, the historical intelligence information source credibility scores, historical similar intelligence information source information, historical intelligence event vectors and historical similar intelligence event vectors of multiple historical intelligence information are processed to determine the sample historical intelligence information accuracy scores of multiple historical intelligence information; Determining expert historical intelligence information accuracy scores for multiple historical intelligence information; Determine a training loss function of an intelligence information accuracy assessment model based on the sample historical intelligence information accuracy score, the expert historical intelligence information accuracy score, the historical intelligence information source credibility score, the historical similar intelligence information source credibility score, the historical intelligence event vector, and the historical similar intelligence event vector; Training the intelligence information accuracy assessment model according to the training loss function to obtain a trained intelligence information accuracy assessment model; Determining a training loss function of an intelligence information accuracy assessment model according to the accuracy score of the sample historical intelligence information, the accuracy score of the expert historical intelligence information, the credibility score of the historical intelligence information source, the credibility score of the historical similar intelligence information source, the historical intelligence event vector, and the historical similar intelligence event vector includes: According to the formula Determine the training loss function of the intelligence information accuracy assessment model ,in, Score the accuracy of the expert historical intelligence information for the i-th historical intelligence information, Score the accuracy of the sample historical intelligence information for the i-th historical intelligence information, is the time when the historical intelligence event of the i-th historical intelligence information occurred, is the location of the historical intelligence event of the i-th historical intelligence information, is the person related to the historical intelligence event of the i-th historical intelligence information, is the historical intelligence event vector of the i-th historical intelligence information, for The transposed vector of is the time of occurrence of the historical similar intelligence event of the historical similar intelligence information of the jth information source of the i-th historical intelligence information, is the location of the historical similar intelligence event of the historical similar intelligence information of the jth information source of the i-th historical intelligence information, is the person related to the historical similar intelligence event of the event of the historical similar intelligence information of the jth information source of the i-th historical intelligence information, Score the credibility of the historical similar intelligence information source of the j-th information source of the i-th historical intelligence information, is the historical similar intelligence event vector of the historical similar intelligence information of the jth information source of the i-th historical intelligence information, is the credibility score of the historical intelligence information source of the i-th historical intelligence information, m is the number of information sources, n is the number of historical intelligence information, i≤n, j≤m, i, j, m and n are positive integers.
2. The method for evaluating the utility of intelligence information based on artificial intelligence according to claim 1, characterized in that: Determine the timeliness score of the intelligence information according to the release time of the intelligence information, the update speed of the intelligence information, and the update frequency of the intelligence information, including: According to the formula Determine the timeliness rating of intelligence information ,in, , and is the preset weight, The time when the intelligence event occurred. The time for intelligence information release. For the speed of intelligence information update, The frequency of intelligence information updates, is the preset hysteresis threshold, To preset the intelligence information update speed threshold, Set the frequency threshold for updating intelligence information.
3. The intelligence information utility evaluation method based on artificial intelligence according to claim 1 is characterized in that: Determine the credibility score of the historical intelligence information source based on the historical intelligence information source information, including: Determine the publisher information of the historical intelligence information according to the source information of the historical intelligence information, wherein the publisher information of the historical intelligence information includes: the accuracy score of the publisher's historical published information and the number of times the publisher has published historical information; Determine the name of the source of the historical intelligence information according to the source information of the historical intelligence information; Obtain the names of known news release agencies, known media, and known academic institutions; Determine the identification result of the message issuing agency according to the name of the source of the historical intelligence information and the name of the known message issuing agency; Determining a media identification result according to the historical intelligence information source name and the known media name; Determine an academic institution identification result according to the name of the historical intelligence information source and the name of the known academic institution; The credibility score of the source of historical intelligence information is determined based on the publisher information of the historical intelligence information, the identification result of the message publishing organization, the media identification result and the identification result of the academic institution.
4. The method for evaluating the utility of intelligence information based on artificial intelligence according to claim 3 is characterized in that: Determining the credibility score of the source of historical intelligence information according to the publisher information of the historical intelligence information, the identification result of the message publishing organization, the media identification result and the identification result of the academic institution includes: According to the formula Determine the credibility score of the historical intelligence information source of the i-th historical intelligence information ,in, and is the preset weight, is the identification result of the news release agency of the i-th historical intelligence information, is the media recognition result of the i-th historical intelligence information, is the academic institution identification result of the i-th historical intelligence information, is the preset release count threshold. Score the accuracy of historical information released by the publisher of the i-th historical intelligence information for the publisher of the k-th information released, To preset the accuracy score threshold, , , , K is the number of times the publisher has published information in history, k≤K, k and K are both positive integers.
5. An artificial intelligence-based intelligence information utility evaluation system for executing the method according to any one of claims 1 to 4, characterized in that: include: A relevant data acquisition module is used to acquire relevant data of intelligence information, wherein the relevant data includes: intelligence information release time, intelligence information update speed, intelligence information update frequency and intelligence information source information; A key entity module is used to determine the key entities of the intelligence information event based on the intelligence information, wherein the key entities include: the time when the intelligence event occurred, the location where the intelligence event occurred, and the persons related to the intelligence event; A similar intelligence information module is used to determine similar intelligence information of multiple information sources of the intelligence information based on the intelligence information; A similar key entity module is used to obtain similar key entities of the event of the similar intelligence information, wherein the similar key entities include: the time when the similar intelligence event occurred, the location where the similar intelligence event occurred and the people related to the similar intelligence event, and determine the similar intelligence information source information of the similar intelligence information; A timeliness scoring module, used to determine the timeliness score of the intelligence information according to the release time of the intelligence information, the update speed of the intelligence information, and the update frequency of the intelligence information; A training model module is used to obtain multiple historical intelligence information and train an intelligence information accuracy assessment model based on the historical intelligence information to obtain a trained intelligence information accuracy assessment model; An accuracy scoring module, used to process the similar intelligence information source information, the intelligence information source information, the key entity and the similar key entity through a trained intelligence information accuracy assessment model to determine an accuracy score of the intelligence information; The utility scoring module is used to determine the utility score of the intelligence information based on the timeliness score and the accuracy score.
