Artificial intelligence policy and regulation management and case library retrieval system

Abstract

The invention provides an artificial intelligence governance policy and regulation and case library retrieval system which is characterized by comprising a data crawling module used for crawling law and policy document data, related to artificial intelligence, of all countries from existing websites of all countries; the data processing module is used for sequentially carrying out preprocessing and data extraction on the legal and policy document data to generate database data; the database module is used for storing database data; and the retrieval module is used for selecting corresponding data from all database data as a retrieval result according to a query statement input by a user. In a word, the system can obtain comprehensive legal policy documents related to artificial intelligence, and efficient retrieval is carried out according to user instructions.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval technology, and in particular to an artificial intelligence governance policy and regulation and case library retrieval system. Background Art

[0003] Although these new AI technologies have broader application prospects, there is an urgent need to regulate various AI risks in terms of social economy, law, and ethics.

[0004] Therefore, in response to the issue of AI security, the world is exploring AI risk governance and policy regulations, and actively discussing the impact of various AI risks on social economy, law, ethics and other aspects.

[0005] Therefore, in the field of AI supervision, it is urgent to establish a publicly searchable global AI governance policy and regulatory system to provide a basis for decision-making and layout in the field of AI risk governance. Summary of the invention

[0006] The present invention is made to solve the above-mentioned problems, and its purpose is to provide an artificial intelligence governance policy and regulation and case library retrieval system.

[0007] The present invention provides an artificial intelligence governance policy, regulation and case library retrieval system, which has the following characteristics: a data crawling module, which is used to crawl legal policy document data related to artificial intelligence in various countries from existing websites of various countries; a data processing module, which is used to pre-process and extract data on legal policy document data in turn to generate database data; a database module, which is used to store database data; a retrieval module, which is used to select corresponding data from all database data as retrieval results according to a query statement input by a user, wherein the data crawling module includes: a static web page crawling unit, which is used to obtain the HTML source code of a static web page in an existing website, and extract the legal policy document data from the HTML source code; a dynamic web page crawling unit, which is used to obtain the legal policy document data according to the internal json change data when the dynamic web page in the existing website is returned.

[0008] The artificial intelligence governance policy, regulation and case library retrieval system provided by the present invention may also have the following features: wherein, the process of the static web page crawling unit acquiring legal policy document data comprises the following steps: step S1, generating request data according to preset search keywords and sending it to an existing website; step S2, parsing the HTTP response data sent by the existing website according to the request data through the BeautifulSoup class to generate a Beautiful Soup object; step S3, extracting data from the BeautifulSoup object to obtain legal policy document data.

[0009] The artificial intelligence governance policies, regulations and case library retrieval system provided by the present invention may also have the following features: In step S3, the static web page crawling unit checks whether there is a "next page" button on the search results page. If so, it turns the page and uses the data extracted on the next page and the data extracted on this page as the content of the same legal policy document data.

[0010] The artificial intelligence governance policy, regulation and case library retrieval system provided by the present invention may also have the following features: wherein, the data crawling module performs incremental crawling according to the Scrapy framework to obtain the legal policy document data updated by each existing website.

[0011] The artificial intelligence governance policy, regulation and case library retrieval system provided by the present invention may also have the following features: the Scrapy framework includes an engine, a scheduler, a downloader, a parser and a pipeline component. The downloader is used to connect servers through the Twisted asynchronous network, obtain web page data, process HTTP requests, process Cookies and process redirections, and return HTTP responses to the engine.

[0012] The artificial intelligence governance policies, regulations and case library retrieval system provided by the present invention may also have the following features: the retrieval module and the database module implement data transmission through AJAX asynchronous interaction technology and JSON data exchange format, and the database module stores database data through the object-relationship ORM of the Django framework.

[0013] The artificial intelligence governance policies, regulations and case library retrieval system provided by the present invention may also have the following features: the legal policy document data includes the title, introduction and text content of artificial intelligence-related legal policy documents.

[0014] The artificial intelligence governance policy, regulation and case library retrieval system provided by the present invention may also have the following features: wherein the retrieval module includes: a precise retrieval unit, which is used to determine whether the query statement is completely matched with the database data, and to use the database data that completely matches the query statement as the retrieval result; a fuzzy retrieval unit, which is used to generate input query features based on the query statement, and to calculate the similarity between the input query features and the feature vectors of each database data, and to select the database data corresponding to multiple feature vectors with high similarity as the retrieval result.

[0015] The artificial intelligence governance policies, regulations and case library retrieval system provided by the present invention may also have the following features: wherein the fuzzy retrieval unit stores a similarity threshold, and the database data corresponding to the feature vector whose similarity is greater than the similarity threshold is the retrieval result.

[0016] Functions and Effects of the Invention

[0017] According to the artificial intelligence governance policies, regulations and case library retrieval system involved in the present invention, on the one hand, the data crawling module adopts targeted content acquisition methods for different web page types, which improves the data acquisition capability and makes the stored data more complete and comprehensive; on the other hand, the retrieval module realizes precise retrieval and fuzzy retrieval, accurately mines data closely related to user needs from the database, and improves the practicality of the system. Therefore, the artificial intelligence governance policies, regulations and case library retrieval system of the present invention can obtain comprehensive legal policy documents related to artificial intelligence, and perform efficient retrieval according to user instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a block diagram of an artificial intelligence governance policy and regulation and case library retrieval system in an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of a process for a static webpage crawling unit to obtain legal policy document data in an embodiment of the present invention;

[0020] Figure 3 Schematic diagram of recall and precision results in an embodiment of the present invention;

[0021] Figure 4 It is a schematic diagram of response time results in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the artificial intelligence governance policies and regulations and case library retrieval system of the present invention.

[0023] This embodiment provides an artificial intelligence governance policy and regulation and case library retrieval system for collecting artificial intelligence-related policies and regulations from various countries and providing them to users for retrieval and query.

[0024] Figure 1 It is a block diagram of an artificial intelligence governance policy, regulation and case library retrieval system in an embodiment of the present invention.

[0025] like Figure 1 As shown, the artificial intelligence governance policies, regulations and case library retrieval system 100 includes a data crawling module 10, a data processing module 20, a database module 30, a retrieval module 40 and a control module 50 for controlling the operation of the above modules.

[0026] The data crawling module 10 is used to crawl legal policy document data related to artificial intelligence in various countries from existing websites of various countries. Among them, the legal policy document data includes the title, introduction and text content of legal policy documents related to artificial intelligence. In this embodiment, the existing websites of various countries are the existing websites of various countries, including the official websites of various countries for publishing policies and regulations.

[0027] Among them, the data crawling module 10 performs incremental crawling according to the Scrapy framework to obtain the legal policy document data updated by each existing website. The Scrapy framework includes an engine, a scheduler, a downloader, a parser and a pipeline component. The downloader is used to connect between servers through the Twisted asynchronous network, obtain web page data, process HTTP requests, process Cookies and process redirections, and return the HTTP response to the engine.

[0028] In this embodiment, a search strategy is set in the Scrapy framework to crawl the new target URL to achieve data acquisition similar to "customization", and a time-based crawling strategy is adopted. By setting the initial time and comparing it with the local time, the latest text update from the set time to the present can be achieved. In addition, the newly crawled text is processed by the data processing module 20 and integrated and saved with the corresponding data in the database module 30, thereby realizing dynamic update of data, improving the efficiency of crawling, and reducing the waste of resources and the amount of data crawled repeatedly.

[0029] The data crawling module 10 includes a static webpage crawling unit 101 and a dynamic webpage crawling unit 102 .

[0030] The static webpage crawling unit 101 is used to obtain the HTML source code of the static webpage in the existing website, and extract the legal policy document data from the HTML source code.

[0031] Figure 2 It is a schematic diagram of the process of a static webpage crawling unit acquiring legal policy document data in an embodiment of the present invention.

[0032] like Figure 2 As shown, the process of the static webpage crawling unit 101 acquiring legal policy document data includes the following steps:

[0033] Step S1, generating request data according to preset search keywords and sending it to an existing website.

[0034] In this embodiment, the static webpage crawling unit 101 constructs a specific URL, i.e., request data, according to the preset search keyword, and sends a request to the target website using an HTTP GET request. In this embodiment, the request uses a timeout parameter and sets the request timeout to 10 seconds to prevent the program from being blocked due to a long wait for a response.

[0035] Step S2, parsing the HTTP response data sent by the existing website according to the request data through the BeautifulSoup class to generate a Beautiful Soup object.

[0036] In this embodiment, the HTTP response data is the HTML content of the search result page.

[0037] Step S3, extracting data from the Beautiful Soup object to obtain legal policy document data.

[0038] Among them, in step S3, the static web page crawling unit checks whether there is a "next page" button on the search results page. If so, it turns the page and uses the data extracted from the next page and the data extracted from this page as the content of the same legal policy document data until there is no next page.

[0039] In this embodiment, the legal policy document data is in the form of a dictionary list. During the parsing process, the function searches for a div tag with a class attribute of SearchResult in HTML, which contains the relevant content of each piece of regulatory information. For each search result, the function searches for the title, i.e., the a tag under the h2 tag, and the URL, and extracts them. The title and URL are encapsulated in a dictionary form and added to a list.

[0040] The dynamic webpage crawling unit 102 is used to obtain legal policy document data according to the internal json change data when the dynamic webpage in the existing website is returned.

[0041] The data processing module 20 is used to perform pre-processing and data extraction on the legal policy document data in sequence to generate database data.

[0042] The database module 30 is used to store database data.

[0043] The database module 30 stores the database data through the object relational database management (ORM) of the Django framework. In this embodiment, Navicat Premium 16 is used as an auxiliary tool for database management and development, and indexes are designed for multidimensional data tables according to the content to be crawled at each layer, thereby realizing the associated query of the database.

[0044] The search module 40 is used to select corresponding data from all database data as search results according to the query statement input by the user.

[0045] In this embodiment, the search module 40 is provided with third-party libraries of Python language: request, flask and googletrans, so as to realize keyword search and built-in translation functions, and can further translate the regulatory title into the language selected by the user, so as to facilitate the user to better understand the search results.

[0046] The search module 40 includes a precise search unit 401 and a fuzzy search unit 402 .

[0047] The precise search unit 401 is used to determine whether the query statement and the database data are completely matched, and to use the database data that completely matches the query statement as the search result.

[0048] In this embodiment, the precise search unit 401 is provided with a keyword search system search_database_with_regex and search_database_advanced, whose main function is to search the database for legal information that matches the regular expression pattern and return the matching results. The function first connects to the database file through the sqlite3.connect() method and creates a cursor object for executing SQL queries. Next, the function executes the SQL query statement to select the title, link, and text content of all legal information from the database. Use the SELECT statement to select specific fields, execute the cursor.fetchall() method, obtain all matching rows, and store the results in the results variable.

[0049] After obtaining the query results, the function closes the database connection and releases resources. Next, the function traverses the query results and processes each piece of legal information. In the process of traversing the results, for each piece of legal information, its title, link and text content are extracted, and the text content is checked to see if it matches the regular expression pattern.

[0050] For each legal information, first extract the title, link and text content from the results. Then use the re.search() method to apply the regular expression pattern to the text content for matching. If the match is successful, the title is translated into English and the matching result is added to the matching_results list. Finally, the function returns the matching result matching_results, which contains the titles and links of all legal information that match the regular expression pattern.

[0051] The fuzzy retrieval unit 402 is used to generate input query features according to the query statement, and calculate the similarity between the input query features and the feature vectors of each database data, and select the database data corresponding to multiple feature vectors with high similarity as the retrieval result.

[0052] The fuzzy search unit 402 stores a similarity threshold, and the database data corresponding to the feature vector with a similarity greater than the similarity threshold is the search result. In this embodiment, the similarity value range is [0.0, 1.0], the similarity threshold is set between 0.7-0.8, and the search results are output in descending order of similarity, so that the user can first confirm more appropriate and more accurate information.

[0053] The retrieval module 40 and the database module 30 implement data transmission through AJAX asynchronous interaction technology and JSON data exchange format.

[0054] The control module 50 stores a control program for controlling the operation of each module.

[0055] In this embodiment, the artificial intelligence governance policy and regulations and case library retrieval system 100 is subjected to retrieval testing, and four tests are performed according to different retrieval purposes, and the corresponding precision, recall and response time are calculated. Among them, precision = number of relevant documents / total number of retrieved documents, recall = number of relevant documents retrieved / all relevant documents in the document library.

[0056] Figure 3 It is a schematic diagram of the recall rate and precision rate results in an embodiment of the present invention.

[0057] like Figure 3 As shown, the horizontal axis is the corresponding test, and the vertical axis is the probability value. It can be seen that the recall rate of the artificial intelligence governance policy and regulations and case library retrieval system 100 in the four tests is above 99%, and the precision rate is above 96%.

[0058] Figure 4 It is a schematic diagram of response time results in an embodiment of the present invention.

[0059] like Figure 4 As shown, the horizontal axis represents each test, namely, test 1, test 2, test 3 and test 4, and the vertical axis represents the response time, namely, the time to return the result, in milliseconds. Figure 4 Each test in the figure corresponds to a circle, and the size of the circle is the number of test results in the corresponding test, and the number of test results is marked around the circle, that is, the number of test results in test 1, test 2, test 3, and test 4 is 163, 604, 1439, and 1916 respectively. In addition, the value corresponding to the center of the circle on the ordinate is the response time of the corresponding test. It can be seen that the response time of the four tests is in the millisecond level, that is, the system has an excellent response speed to query requests.

[0060] Functions and Effects of the Embodiments

[0061] According to the AI ​​governance policy, regulation and case library retrieval system involved in this embodiment, on the one hand, the data crawling module adopts targeted content acquisition methods for different web page types, which improves the data acquisition capability and makes the stored data more complete and comprehensive; on the other hand, the retrieval module realizes precise retrieval and fuzzy retrieval, accurately mines data closely related to user needs from the database, and improves the practicality of the system. In short, this method can obtain comprehensive legal policy documents related to AI and perform efficient retrieval according to user instructions.

[0062] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An artificial intelligence governance policy and regulations and case library retrieval system, characterized by: include: The data crawling module is used to crawl legal policy document data related to artificial intelligence in various countries from existing websites of various countries; A data processing module, used to perform pre-processing and data extraction on the legal policy document data in sequence to generate database data; A database module, used for storing the database data; A search module is used to select corresponding data from all the database data as search results according to the query statement input by the user. Wherein, the data crawling module includes: A static webpage crawling unit, used for obtaining the HTML source code of the static webpage in the existing website, and extracting the legal policy document data from the HTML source code; The dynamic webpage crawling unit is used to obtain the legal policy document data according to the internal json change data when the dynamic webpage in the existing website is returned.

2. The artificial intelligence governance policy and regulations and case library retrieval system according to claim 1, Features: The process of the static webpage crawling unit acquiring the legal policy document data includes the following steps: Step S1, generating request data according to preset search keywords and sending the request data to the existing website; Step S2, parsing the HTTP response data sent by the existing website according to the request data through the BeautifulSoup class to generate a Beautiful Soup object; Step S3, extracting data from the Beautiful Soup object to obtain the legal policy document data.

3. The artificial intelligence governance policy, regulations and case library retrieval system according to claim 2 is characterized by: in, In step S3, the static web page crawling unit checks whether there is a "next page" button on the search result page. If so, it turns the page and uses the data extracted from the next page and the data extracted from this page as the content of the same legal policy document data.

4. The artificial intelligence governance policy, regulations and case library retrieval system according to claim 1 is characterized by: in, The data crawling module performs incremental crawling according to the Scrapy framework to obtain the legal policy document data updated by each of the existing websites.

5. The artificial intelligence governance policy, regulation and case library retrieval system according to claim 4 is characterized by: in, The Scrapy framework includes engine, scheduler, downloader, parser and pipeline components. The downloader is used to connect servers through the Twisted asynchronous network, obtain web page data, process HTTP requests, process Cookies and process redirection, and return HTTP responses to the engine.

6. The artificial intelligence governance policy, regulations and case library retrieval system according to claim 1 is characterized by: in, The retrieval module and the database module realize data transmission through AJAX asynchronous interaction technology and JSON data exchange format. The database module stores the database data through the object-relationship ORM of the Django framework.

7. The artificial intelligence governance policy, regulation and case library retrieval system according to claim 1 is characterized by: in, The legal policy document data includes the title, introduction and text content of artificial intelligence-related legal policy documents.

8. The artificial intelligence governance policy, regulations and case library retrieval system according to claim 1, Features: Wherein, the retrieval module includes: An accurate search unit, used to determine whether the query statement is completely matched with the database data, and to use the database data that completely matches the query statement as the search result; The fuzzy retrieval unit is used to generate an input query feature according to the query statement, calculate the similarity between the input query feature and the feature vectors of each database data, and select the database data corresponding to multiple feature vectors with high similarity as the retrieval result.

9. The artificial intelligence governance policy, regulation and case library retrieval system according to claim 8 is characterized by: in, The fuzzy retrieval unit stores a similarity threshold, and the database data corresponding to the feature vector having a similarity greater than the similarity threshold is the retrieval result.

Images