Big data processing and analysis system based on artificial intelligence

Abstract

The present invention relates to the technical field of big data analysis, and in particular to a big data processing and analysis system based on artificial intelligence, comprising a user information collection module, an information input module, a data retrieval module, a personality analysis module, a content recognition module, and a terminal recognition module. The system determines the information component of input matching information by retrieving information according to a user's target, and then determines the pre-adaptation information of the target user based on historical browsing information by taking the information component as basic data of a user portrait. After that, the information product appearing in the pre-adaptation information is recognized, and then the information value of the pre-adaptation information is determined based on the information product existing in the pre-adaptation information. According to the information value, the pre-adaptation information is divided into substantial matching information and virtual effect information, and the substantial matching information is displayed on a terminal device, so as to ensure the content quality of the user's search result, improve the information search efficiency, and thus enhance the user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of big data analysis, and in particular to a big data processing and analysis system based on artificial intelligence. Background Art

[0002] "Big data" is defined as "massive, high-growth, diversified information assets that need to be processed to generate stronger decision-making power, insight discovery, and process optimization capabilities." Its strategic significance is not only the mastery of big data information technology, but more importantly, how to professionally analyze and process this massive information.

[0003] The prior art CN111753170A discloses a big data rapid retrieval system and method. The present invention first collects first verification information through the user terminal after obtaining authorization from the user terminal, then performs retrieval according to the first information retrieval request, establishes a user portrait corresponding to the first verification information based on the retrieved first target information, and then collects second verification information through the user terminal when a second information retrieval request is obtained, and determines the consistency between the second verification information and the first verification information, and finally determines whether to perform information retrieval based on the user portrait based on the consistency judgment result between the second verification information and the first verification information.

[0004] However, with the development of the Internet era, many merchants or bloggers like to promote their products online, which leads to a large amount of duplicate information or advertising products in the search results when users search for information online. This will cause visual interference to users, making it impossible for users to directly judge the compatibility of the information with themselves, making it difficult for users to find the information they need, thereby reducing the user experience. Summary of the invention

[0005] The purpose of the present invention is to solve the problems in the background technology and to propose a big data processing and analysis system based on artificial intelligence.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] Big data processing and analysis system based on artificial intelligence, including:

[0008] The user information collection module is used to collect the historical browsing information of the target user and transmit it to the personality analysis module;

[0009] An information input module is used to generate target retrieval information through a terminal device and transmit it to a data retrieval module;

[0010] A data retrieval module is used to retrieve the target retrieval information in the information database to obtain input matching information, and then the data retrieval module transmits the input matching information to the personality analysis module;

[0011] The personality analysis module is used to perform front-end interference processing on input matching information based on the target user's historical browsing information, including:

[0012] Perform information recognition on the input matching information and generate information tags. At the same time, all information tags in an input matching information are used as information elements to obtain a matching tag set. Information decomposition is performed on the target retrieval information to obtain key information phrases. The key information phrases are used as semantic backgrounds to identify feature categories of information elements. Based on the number of times the feature categories appear in all matching tag sets, feature ratios are calculated, and information components are determined based on the feature ratios.

[0013] Determine the explicit component factors of target users based on information components and historical browsing information;

[0014] The explicit component factor is used as a concrete matching parameter, and the input matching information is matched again, and the pre-adaptation information is determined, and the personality analysis module transmits the pre-adaptation information to the content recognition module;

[0015] The content recognition module is used to identify the products that appear in the pre-adaptation information, mark the pre-adaptation information where the products appear as secondary operation information, and mark the products that appear as information products, then intercept the product fragment information of the information products, and then calculate the target occurrence frequency and text similarity of the same information product in the secondary operation information, and then calculate the information value of the secondary operation information in combination with the number of information products in the secondary operation information, and finally divide the secondary operation information into substantial matching information and virtual effect information according to the information value;

[0016] The terminal identification module is used to identify the actual pairing information and display the actual pairing information on the terminal device of the target user.

[0017] As a further solution of the present invention, a method for identifying a characteristic category of an information element includes:

[0018] Identify the information appearing in the input matching information and generate information tags, take all the information tags in the input matching information as information elements, and form a matching tag set, where the input matching information corresponds to the matching tag set one by one;

[0019] Decomposing the target retrieval information. Decomposing the target retrieval information means to analyze the semantics of the target retrieval information and extract key information phrases. Key information phrases refer to phrases that have actual characteristic directions in their meanings, including nouns and verbs.

[0020] Identify and process the information elements in all matching tag sets. If there is an information element that is consistent with the key information phrase, delete the information element from the corresponding matching tag set.

[0021] Taking key information phrases as semantic background, the classification task-related algorithms in natural language processing technology are used to identify the lexical categories of information elements, and the identified lexical categories are marked as feature categories, where lexical categories refer to the classification of vocabulary according to certain standards and rules.

[0022] As a further solution of the present invention, the method for determining the information component according to the characteristic ratio comprises:

[0023] Taking a feature category as the target feature category, judging whether the target feature category appears in the matching tag set, and if the target feature category appears in the matching tag set, setting a mark in the corresponding tag set;

[0024] Count the number of matching tag sets with marked marks to obtain the marked value, and then divide the marked value by the total number of matching tag sets to obtain the feature ratio;

[0025] The feature ratio is compared with the ratio threshold. If the feature ratio is less than the ratio threshold, the target feature category is marked as an irrelevant feature. Conversely, if the feature ratio is greater than or equal to the ratio threshold, the target feature category is marked as an information component.

[0026] As a further solution of the present invention, a method for determining the dominant component factor of a target user includes:

[0027] Taking information component as the basic attribute of user portrait, the convolutional neural network model is used to perform deep feature extraction on historical browsing information, and then the user portrait of the target user is determined, where the elements in the user portrait are the explicit component factors of the target user.

[0028] As a further solution of the present invention, a method for determining the pre-adaptation information includes:

[0029] The explicit component factor in the information component is used as a concrete matching parameter, and the information elements in the matching tag set are identified:

[0030] First, identify the information element corresponding to the information component in the matching label set and mark it as a personal element. Perform consistency identification on the personal element and the explicit component factor corresponding to its information component. If the personal elements in the matching label set are all the same as the corresponding explicit component factors, the corresponding input matching information is marked as pre-adaptation information. Conversely, if any personal element in the matching label set is different from the explicit component factor, the input matching information corresponding to this matching label set is not marked.

[0031] As a further solution of the present invention, a method for calculating the target occurrence frequency includes:

[0032] Based on visual recognition technology, the products appearing in the pre-adaptation information are identified. If a product appears in the pre-adaptation information, the corresponding pre-adaptation information is marked as secondary operation information, and the product in the secondary operation information is marked as information product i, i∈[1,I], indicating that there are I information products in a piece of secondary operation information;

[0033] Arbitrarily select an information product as the target product i, based on the formula The target occurrence frequency fi is obtained, where the target occurrence frequency fi refers to the frequency at which the target product i appears in the secondary operation information, Ni represents the number of secondary operation information containing the target product i, and Ns represents the total number of secondary operation information.

[0034] As a further solution of the present invention, a method for calculating text similarity includes:

[0035] According to the information product i in the secondary operation information, the relevant contents of the secondary operation information and the information product i are segmented and marked as product segment information, wherein in the secondary operation information, the information product i and the product segment information correspond to each other;

[0036] Extract product fragment information of target product i from all secondary operation information, and pre-process the extracted product fragment information to unify the information format of the product fragment information and mark it as the same product information. The information format includes text format, image format, audio format and video format;

[0037] Based on the Word2Vec model in the neural network model, the same information is separated into phrases and the Word2Vec vector of each phrase is calculated. Then, in a piece of same information, the mean of the Word2Vec vectors of all phrases is taken and used as the text vector of this piece of same information.

[0038] The cosine similarity algorithm is used to calculate the similarity of the text vectors of all the same-product information, and the calculation result is marked as the text similarity value WXi of the target product i.

[0039] As a further solution of the present invention, a method for determining the actual pairing information and the virtual role information includes:

[0040] Using the formula The information value XE of the secondary operation information is obtained, where I represents the total information product in the secondary information, DS represents the information content of the secondary operation information, β1 and β2 are both proportional coefficients, b1 and b2 are fixed factors, and 0<b1<1, 0<b2<1, c1 is a constant coefficient, fi and WXi represent the target occurrence frequency and text similarity value of the target product i respectively;

[0041] The information value XE is compared with the information threshold Xy. If XE≥Xy, the corresponding secondary operation information is marked as actual pairing information. Otherwise, if XE<Xy, the corresponding secondary operation information is marked as virtual action information.

[0042] As a further solution of the present invention, if the product does not appear in the pre-adaptation information, the corresponding pre-adaptation information is directly marked as the actual pairing information.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] The present invention determines the corresponding input matching information according to the user's target search information, then identifies the vocabulary category in the input matching information, determines the information component of the input matching information, and then uses the information component as the basic data of the user portrait to determine the specific user portrait of the target user in the historical browsing information. According to the specific user portrait of the target user, the pre-adaptation information is determined in the input matching information, thereby ensuring that the search results are highly correlated with the target search information, and can improve the accuracy of keyword matching;

[0045] The present invention identifies the information products appearing in the pre-adaptation information, calculates the text similarity values ​​of the information products and the frequencies of the information products, and then determines the information value of the pre-adaptation information based on the information products existing in the pre-adaptation information. Then, according to the information value, the pre-adaptation information is divided into substantial matching information and virtual effect information, and then the substantial matching information is displayed on the terminal device. On the one hand, the content quality of the user's search results is guaranteed, and on the other hand, the information retrieval efficiency is improved, thereby enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0048] Reference Figure 1 , a big data processing and analysis system based on artificial intelligence, including user information collection module, information input module, data retrieval module, personality analysis module, content recognition module, and terminal recognition module;

[0049] The user information collection module is used to collect the browsing history of the target user on the terminal device and generate historical browsing information. The user information collection module then transmits the historical browsing information to the personality analysis module. The target user refers to the user of the terminal device, and the terminal device includes electronic devices such as computers and mobile phones.

[0050] The information input module is used to generate target retrieval information. The target user inputs the data information to be queried through the terminal device. The information input module marks the data information to be queried input by the target user as the target retrieval information and transmits it to the data retrieval module.

[0051] The data retrieval module is used to receive target retrieval information and search the target retrieval information in the information database to obtain input matching information. After that, the data retrieval module transmits the input matching information to the personality analysis module, wherein the input matching information includes text information and video information;

[0052] The personality analysis module is used to perform front-end interference processing on the input matching information according to the historical browsing information of the target user. The specific front-end interference processing method includes:

[0053] S1: Determine the information components based on the target retrieval information and the input matching information:

[0054] Identify the information appearing in the input matching information and generate information tags, use all the information tags in the input matching information as information elements, and form a matching tag set, wherein the input matching information corresponds to the matching tag set one by one;

[0055] Then, the target retrieval information is disassembled. Information disassembly is to perform semantic analysis on the target retrieval information and extract key information phrases. Key information phrases refer to phrases that have actual feature orientations in their meanings, including nouns and verbs.

[0056] Identify and process the information elements in all matching tag sets. If there is an information element that is consistent with the key information phrase, delete the information element from the corresponding matching tag set. After all information elements are identified and processed, the information elements in each matching tag set are the characteristic information tags in the corresponding input matching information.

[0057] Then, the key information phrase is used as the semantic background, and the classification task-related algorithm in the natural language processing technology is used to identify the vocabulary category of the information element, and the identified vocabulary category is marked as a feature category, where the vocabulary category refers to the classification of vocabulary according to certain standards and rules. For example, if the information element contains female and tall, after the vocabulary category of the information element is identified, the vocabulary category of female corresponds to gender, and the vocabulary category of tall corresponds to height;

[0058] It should be further explained that the purpose of using the key information phrase as the semantic background is to limit the semantic scene of the information element, thereby improving the accuracy of vocabulary category recognition, and the classification task-related algorithm in this embodiment uses the random forest algorithm. The specific random forest algorithm is an existing technology and will not be described in detail here;

[0059] A feature category is used as a target feature category, and it is determined whether the target feature category appears in the matching tag set. If the target feature category appears in the matching tag set, a mark is set in the corresponding tag set. In this embodiment, the mark is set to 1.

[0060] Count the number of matching tag sets with marked marks to obtain the marked value, and then divide the marked value by the total number of matching tag sets to obtain the feature ratio;

[0061] The feature ratio is compared with the ratio threshold. If the feature ratio is less than the ratio threshold, the target feature category is marked as an irrelevant feature. On the contrary, if the feature ratio is greater than or equal to the ratio threshold, the target feature category is marked as an information component. The ratio threshold is obtained by a person skilled in the art after big data calculation.

[0062] All feature categories are taken as target feature categories in turn, and processed according to the above method to obtain different information components;

[0063] S2: Determine the explicit component factor of the target user in the historical browsing information based on the information component;

[0064] Taking information component as the basic attribute of user portrait, the convolutional neural network model is used to perform deep feature extraction on historical browsing information, and then determine the user portrait of the target user, where the elements in the user portrait are the explicit component factors of the target user. For example, the information component includes gender and dressing style. According to the historical browsing information of the target user, it is determined that the gender of the target user is female and the dressing style is casual. In this case, female is the explicit component factor of the gender information component, and casual style is the explicit component factor of the dressing style.

[0065] Furthermore, using a convolutional neural network to perform deep feature extraction on the target user's historical browsing information is an existing technology, which will not be described in detail here;

[0066] S3: identifying the pre-adaptation information in the input matching information according to the explicit component factor in the information component;

[0067] The explicit component factors in the information components are used as concrete matching parameters, and the information elements in the matching tag set are identified: first, the information elements corresponding to the information components in the matching tag set are identified, and marked as individual elements, and the individual elements and the explicit component factors corresponding to the information components are identified for consistency. If the individual elements in the matching tag set are all the same as the corresponding explicit component factors, the corresponding input matching information is marked as pre-adaptation information. On the contrary, if any individual element in the matching tag set is different from the explicit component factor, the input matching information corresponding to this matching tag set is not marked;

[0068] All matching tag sets are processed in the above manner, and then the input matching information is screened to obtain pre-adaptation information, which is the information retrieval result that is closest to the user profile of the target user among the input matching information that matches the target search information;

[0069] The personality analysis module then transmits the pre-adaptation information to the content recognition module;

[0070] The content identification module is used to analyze the information value in the pre-adaptation information, and determine the content nature of the pre-adaptation information based on the information value, wherein the pre-adaptation information is divided into substantial matching information and virtual effect information according to the content nature. The specific method of analyzing the information value of the pre-adaptation information includes:

[0071] Based on visual recognition technology, the products appearing in the pre-adaptation information are identified. If the product does not appear in the pre-adaptation information, the corresponding pre-adaptation information is directly marked as the actual pairing information. If the product appears in the pre-adaptation information, the corresponding pre-adaptation information is marked as the secondary operation information. At the same time, the product in the secondary operation information is marked as information product i, i∈[1,I], indicating that there are I information products in a piece of secondary operation information. It should be further explained that there may be only one product or multiple products in a piece of pre-adaptation information.

[0072] According to the information product i in the secondary operation information, the relevant contents of the secondary operation information and the information product i are segmented and marked as product segment information, wherein in the secondary operation information, the information product i and the product segment information correspond to each other;

[0073] Arbitrarily select an information product as the target product a, where a∈i, based on the formula Obtain the target occurrence frequency fa, where the target occurrence frequency fa refers to the frequency of the target product a appearing in the secondary operation information, Na represents the number of secondary operation information containing the target product a, and Ns represents the total number of secondary operation information;

[0074] Extract product fragment information of the target product a from all secondary operation information, and pre-process the extracted product fragment information to unify the information format of the product fragment information and mark it as the same product information. In this embodiment, optical character recognition technology and large language model technology are used to uniformly convert the information format of the extracted product fragment information into a text format, and the information format includes text format, image format, audio format and video format. The optical character recognition technology and large language model technology are prior arts for converting the information format, which will not be described in detail here.

[0075] First, based on the Word2Vec model in the neural network model, the same information is separated into phrases and the Word2Vec vector of each phrase is calculated. Then, in a piece of same information, the mean of the Word2Vec vectors of all phrases is taken and used as the text vector of this piece of same information.

[0076] Using the cosine similarity algorithm, the similarity of the text vectors of all the same-product information is calculated, and the calculation result is marked as the text similarity value WXa of the target product a;

[0077] Among them, the Word2Vec model and the cosine similarity algorithm are both existing technologies and will not be described in detail here;

[0078] All information products are taken as target products in turn, and processed according to the above method to obtain the target occurrence frequency fi and text similarity value WXi of each information product i;

[0079] Then use the formula The information value XE of the secondary operation information is obtained, wherein I represents the total information product in the secondary information, DS represents the information amount of the secondary operation information, β1 and β2 are both proportional coefficients, b1 and b2 are fixed factors, and 0<b1<1, 0<b2<1, c1 is a constant coefficient, and the specific β1, β2, b1, b2 and c1 are obtained by those skilled in the art after multiple operations using a big data model;

[0080] Then, the information value XE is compared with the information threshold Xy. If XE≥Xy, the corresponding secondary operation information is marked as actual pairing information. On the contrary, if XE<Xy, the corresponding secondary operation information is marked as virtual action information. The specific value of the information threshold Xy is obtained by technicians in this field after big data calculation.

[0081] There is a two-way communication connection between the content identification module and the terminal identification module;

[0082] The terminal identification module is used to filter the virtual role information, identify the actual pairing information, and display the actual pairing information on the terminal device of the target user.

[0083] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A big data processing and analysis system based on artificial intelligence, characterized by: include: The user information collection module is used to collect the historical browsing information of the target user and transmit it to the personality analysis module; An information input module is used to generate target retrieval information through a terminal device and transmit it to a data retrieval module; A data retrieval module is used to retrieve the target retrieval information in the information database to obtain input matching information, and then the data retrieval module transmits the input matching information to the personality analysis module; The personality analysis module is used to perform front-end interference processing on input matching information based on the target user's historical browsing information, including: Perform information recognition on the input matching information and generate information tags. At the same time, all information tags in an input matching information are used as information elements to obtain a matching tag set. Information decomposition is performed on the target retrieval information to obtain key information phrases. The key information phrases are used as semantic backgrounds to identify feature categories of information elements. Based on the number of times the feature categories appear in all matching tag sets, feature ratios are calculated, and information components are determined based on the feature ratios. Comparing the feature ratio with the ratio threshold, if the feature ratio is less than the ratio threshold, marking the feature category as an irrelevant feature, otherwise, if the feature ratio is greater than or equal to the ratio threshold, marking the feature category as an information component; Based on the information component and historical browsing information, the explicit component factor of the target user is determined, wherein the specific calculation process includes: Taking information components as the basic attributes of user portraits, the convolutional neural network model is used to perform deep feature extraction on historical browsing information, and then determine the user portrait of the target user, where the elements in the user portrait are the explicit component factors of the target user; The explicit component factor is used as a concrete matching parameter, and the input matching information is matched again, and the pre-adaptation information is determined. The personality analysis module transmits the pre-adaptation information to the content identification module, wherein the method for determining the pre-adaptation information includes: The explicit component factor in the information component is used as a concrete matching parameter, and the information elements in the matching tag set are identified: First, identify the information element corresponding to the information component in the matching label set, and mark it as a personality element, and identify the consistency between the personality element and the explicit component factor corresponding to its information component. If the personality elements in the matching label set are all the same as the corresponding explicit component factors, the corresponding input matching information is marked as pre-adaptation information. On the contrary, if any personality element in the matching label set is different from the explicit component factor, the input matching information corresponding to this matching label set is not marked; The content identification module is used to identify the products appearing in the pre-adaptation information, mark the pre-adaptation information where the products appear as secondary operation information, and mark the products appearing at the same time as information products, then intercept the product fragment information of the information products, and then calculate the target occurrence frequency and text similarity of the same information product in the secondary operation information, and then calculate the information value of the secondary operation information in combination with the number of information products in the secondary operation information, and finally divide the secondary operation information into substantial matching information and virtual effect information according to the information value, wherein the specific determination method of substantial matching information and virtual effect information includes: Using the formula The information value XE of the secondary operation information is obtained, where I represents the total information product in the secondary information, and DS represents the information content of the secondary operation information. and are proportional coefficients, b1 and b2 are fixed factors, and 0<b1<1, 0<b2<1, c1 is a constant coefficient, fi and WXi represent the target occurrence frequency and text similarity value of target product i respectively; Compare the information value XE with the information threshold Xy. If XE≥Xy, the corresponding secondary operation information is marked as actual pairing information. Otherwise, if XE<Xy, the corresponding secondary operation information is marked as virtual action information. The terminal identification module is used to identify the actual pairing information and display the actual pairing information on the terminal device of the target user.

2. The big data processing and analysis system based on artificial intelligence according to claim 1 is characterized in that: Methods for identifying characteristic categories of information elements include: Identify the information appearing in the input matching information and generate information tags, take all the information tags in the input matching information as information elements, and form a matching tag set, where the input matching information corresponds to the matching tag set one by one; Decomposing the target retrieval information. Decomposing the target retrieval information means to analyze the semantics of the target retrieval information and extract key information phrases. Key information phrases refer to phrases that have actual characteristic directions in their meanings, including nouns and verbs. Identify and process the information elements in all matching tag sets. If there is an information element that is consistent with the key information phrase, delete the information element from the corresponding matching tag set. Taking key information phrases as semantic background, the classification task-related algorithms in natural language processing technology are used to identify the lexical categories of information elements, and the identified lexical categories are marked as feature categories, where lexical categories refer to the classification of vocabulary according to certain standards and rules.

3. The big data processing and analysis system based on artificial intelligence according to claim 2 is characterized in that: The calculation method of information component includes: Taking a feature category as the target feature category, judging whether the target feature category appears in the matching tag set, and if the target feature category appears in the matching tag set, setting a mark in the corresponding tag set; The number of matching tag sets with marked marks is counted to obtain the marked value, and then the marked value is divided by the total number of matching tag sets to obtain the feature ratio.

4. The big data processing and analysis system based on artificial intelligence according to claim 1 is characterized in that: The calculation method of target occurrence frequency includes: Based on visual recognition technology, the products appearing in the pre-adaptation information are identified. If a product appears in the pre-adaptation information, the corresponding pre-adaptation information is marked as secondary operation information, and the product in the secondary operation information is marked as information product i, i∈[1,I], indicating that there are I information products in a piece of secondary operation information; Arbitrarily select an information product as the target product i, based on the formula The target occurrence frequency fi is obtained, where the target occurrence frequency fi refers to the frequency at which the target product i appears in the secondary operation information, Ni represents the number of secondary operation information containing the target product i, and Ns represents the total number of secondary operation information.

5. The big data processing and analysis system based on artificial intelligence according to claim 4 is characterized in that: The calculation methods of text similarity include: According to the information product i in the secondary operation information, the relevant contents of the secondary operation information and the information product i are segmented and marked as product segment information, wherein in the secondary operation information, the information product i and the product segment information correspond to each other; Extract product fragment information of target product i from all secondary operation information, and pre-process the extracted product fragment information to unify the information format of the product fragment information and mark it as the same product information. The information format includes text format, image format, audio format and video format; Based on the Word2Vec model in the neural network model, the same information is separated into phrases and the Word2Vec vector of each phrase is calculated. Then, in a piece of same information, the mean of the Word2Vec vectors of all phrases is taken and used as the text vector of this piece of same information. The cosine similarity algorithm is used to calculate the similarity of the text vectors of all the same-product information, and the calculation result is marked as the text similarity value WXi of the target product i.

6. The big data processing and analysis system based on artificial intelligence according to claim 4 is characterized in that: If the product does not appear in the pre-adaptation information, the corresponding pre-adaptation information will be directly marked as the actual pairing information.

Images