Personalized recommendation method based on user collaborative filtering and aggregate statistics

Abstract

The present invention discloses a personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics. The recommendation method is respectively composed of a data preprocessing module, a user similarity calculation module and a secondary sorting module based on significant aggregate statistics. The data preprocessing module processes and pre-calculates the weights of the original user's self-selected stocks and holding stocks, obtains a clean user stock data set and a user sparse vector after processing, and stores them in a warehouse. The present invention relates to the field of data information processing technology. The personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics does not require manual extraction of complex features and manual annotation of data in the process of personalized item recommendation. The advance pre-calculation processing speeds up the performance of the recommendation. The user's behavior data can be reflected in the recommendation results in near real time. The items of interest to the user group that are most similar to the user's current interests can be dynamically recommended, which fully meets the engineering application.

Description

Technical Field

[0001] The present invention relates to the technical field of data information processing, and in particular to a personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics. Background Art

[0002] In personalized recommendations, it is often necessary to find other most similar users based on the user's historical behavior data and a certain metric method, and then recommend other items that they have browsed after sorting them according to user similarity, and then add a certain secondary sorting model to improve CTR. Although there are many recommendation algorithms that can meet the requirements, most algorithms, especially secondary sorting, require a large amount of data to train the model, or require complex feature engineering and other preliminary work. For a small amount of sample data or unbalanced data, the model effect may not be satisfactory; and complex feature engineering will increase the difficulty of model development and affect the project progress.

[0003] A fast, simple and effective personalized user similarity recommendation method can not only accelerate project development progress, but also promote the industry application of artificial intelligence in the recommendation field. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the shortcomings of the prior art, the present invention provides a personalized recommendation method based on user collaborative filtering and aggregate statistics, which solves the problem that most existing algorithms, especially secondary sorting, require a large amount of data to train the model, or require complex feature engineering and other preliminary work. The model effect may not be satisfactory for a small amount of sample data or unbalanced data; and complex feature engineering will increase the difficulty of model development and affect the progress of the project.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics, wherein the recommendation system includes a data preprocessing module, a user similarity calculation module and a secondary sorting module based on significant aggregate statistics, and the recommendation method specifically includes the following steps:

[0008] S1. After the original user selected stocks and holding stocks data are processed by the data preprocessing module, a clean user stock data set and the corresponding user sparse vector are obtained.

[0009] S2. According to the input user and user vector, the cosine similarity distance formula in the user similarity calculation module is used to calculate the user similarity and find the top similar users;

[0010] S3. According to the topk similar users obtained in step S2, secondary sorting is performed through a secondary sorting module based on significant aggregation statistics, and the topM item sorting results are returned.

[0011] Preferably, the method for processing the original user selected stocks and holding stocks data by the data preprocessing module in step S1 specifically comprises the following steps:

[0012] a1. The weighted formula for user selected stocks and holding stocks data and subsequent new feature dimension scores is as follows:

[0013] in

[0014] In formula 1, uss jk represents the interest score of user j in stock k; score i Indicated in USS jk When calculating, the score of the i-th feature, the specific score i The value of is determined by business logic; i Represents the weighted weight of the i-th feature;

[0015] a2. Pre-calculate the normalized eigenvector. From the vector cosine calculation formula, we know that

[0016]

[0017] From formula 2, we can know that the cosine distance calculation pre-calculates the normalized user vector, and the pre-calculation formula is:

[0018]

[0019] Preferably, in step a1, i It is used to weight the scores of each feature.

[0020] Preferably, in step S2, the corresponding user pre-calculated vector is found according to the input user id, and the user similarity score is calculated according to the vector cosine calculation formula to obtain the topK similar users. In this process, the speed is improved by the retrieval capability based on the common database. Since the sparse stock vectors selected by the user's holdings are not reduced, the results are controllable and interpretable.

[0021] Preferably, the common database is one of elasticsearch or milvus.

[0022] Preferably, the secondary sorting method in step S3 specifically includes the following steps:

[0023] b1. Perform significant aggregation on the topK users, then count the stocks purchased by the topK users, and then divide each statistical result by the overall global statistical result to obtain the score of each statistical stock. The formula is as follows:

[0024]

[0025] Among them, card pre (stock l ) means stock l The number of occurrences in the previous data set; card total (stock l ) means stock l The number of occurrences in the backend dataset; card total Indicates the total number of all stocks; card pre_total Represents the total number of stocks in the previous data set;

[0026] b2. Calculate the recommended stock scores. The specific calculation formula is as follows:

[0027] recScore i =cos i ,u * > uss.signif i

[0028] Among them, cos i ,u * > indicates that the similarity vector of the topk users most similar to user i has a dimension of 1×K; uss is the user vector corresponding to the topK users, with a dimension of K×N, where N represents the total number of items; signif i Represents the statistical results of the significant keyword aggregation of user i, with a dimension of 1×N; the '.' multiplication represents the multiplication of the corresponding elements of the vector; recScore i represents the recommended stock score of user i. b3. According to business requirements, just take the top M stocks.

[0029] Preferably, the significant aggregation in step b1 is used to display unusual words of a specific group.

[0030] The present invention also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute a personalized recommendation method based on a combination of user collaborative filtering and aggregate statistics.

[0031] (III) Beneficial effects ​​

[0032] The present invention provides a personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics. Compared with the prior art, it has the following beneficial effects:

[0033] (1) The personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics is composed of a data preprocessing module, a user similarity calculation module and a secondary sorting module based on significant aggregate statistics. The method steps are as follows: first, the original user selected stocks and holding stock data are processed and weights are pre-calculated by the data preprocessing module to obtain a clean user stock data set and user sparse vectors after processing and store them in the database; after the user sparse vector processing is completed, the user similarity calculation module can be generated according to the sparse vector; the stocks held by the topk similar users and the stocks held by the entire user group are sorted based on significant aggregate statistics. In the process of personalized item recommendation, there is no need to manually extract complex features and manually annotate data. The advance pre-calculation processing speeds up the recommendation performance, and the user's behavior data can be reflected in the recommendation results in near real time.

[0034] (2) The personalized recommendation method based on user collaborative filtering and aggregate statistics can dynamically recommend items of interest to the user group that is most similar to the user's current interests, fully meeting engineering applications. The present invention takes the user's self-selected stocks and holdings data as an example to recommend the stocks that are of most interest to the user group that is most similar to the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the present invention;

[0036] Figure 2 A block diagram of the working principle of an electronic device provided by an embodiment of the present invention.

[0037] In the figure: 800 electronic device, 801 memory, 802 processor, 803 bus, 804 communication interface. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1-2The embodiment of the present invention provides a technical solution: a personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics, wherein the recommendation system includes a data preprocessing module, a user similarity calculation module, and a secondary sorting module based on significant aggregate statistics. The recommendation method specifically includes the following steps:

[0040] S1. After the original user selected stocks and holding stocks data are processed by the data preprocessing module, a clean user stock data set and the corresponding user sparse vector are obtained.

[0041] S2. According to the input user and user vector, the cosine similarity distance formula in the user similarity calculation module is used to calculate the user similarity and find the top similar users;

[0042] S3. According to the topk similar users obtained in step S2, secondary sorting is performed through a secondary sorting module based on significant aggregation statistics, and the topM item sorting results are returned.

[0043] In the embodiment of the present invention, the method for processing the original user selected stocks and holding stocks data by the data preprocessing module in step S1 specifically includes the following steps:

[0044] a1. The weighted formula for user selected stocks and holding stocks data and subsequent new feature dimension scores is as follows:

[0045] in

[0046] In formula 1, uss jk represents the interest score of user j in stock k; score i Indicated in USS jk When calculating, the score of the i-th feature is shown in Appendix 1 for specific feature descriptions. i The value of is determined by business logic; i represents the weight of the i-th feature, w i It is used to weight the scores of each feature in Appendix 1;

[0047] a2. Pre-calculate the normalized eigenvector. From the vector cosine calculation formula, we know that

[0048]

[0049] From formula 2, we can know that the cosine distance calculation pre-calculates the normalized user vector, and the pre-calculation formula is:

[0050]

[0051] In the embodiment of the present invention, in step S2, the corresponding user pre-calculated vector is found according to the input user id, and the user similarity score is calculated according to the vector cosine calculation formula of formula 2 to obtain the topK similar users. In this process, the speed is improved by the retrieval capability based on the common database. Since the sparse stock vectors selected by the user's holdings are not reduced, the results are controllable and interpretable. The common database is one of elasticsearch or milvus.

[0052] In the embodiment of the present invention, the secondary sorting method in step S3 specifically includes the following steps:

[0053] b1. Perform significant aggregation on the topK users, and then count the stocks purchased by the topK users. Significant aggregation is also called significant keyword aggregation, which is used to display unusual words for a specific group. Then divide each statistical result by the overall global statistical result to obtain the score of each statistical stock. The formula is as follows:

[0054]

[0055] Among them, card pre (stock l ) means stock l The number of occurrences in the previous dataset (topK user dataset); card total (stock l ) means stock l The number of occurrences in the backend dataset (the overall dataset); card total Indicates the total number of all stocks; card pre_total Represents the total number of stocks in the previous data set;

[0056] b2. Calculate the recommended stock scores. The specific calculation formula is as follows:

[0057] recScore i =cos i ,u * > uss.signif i (Formula 5)

[0058] Among them, cos i ,u * > indicates that the similarity vector of the topk users most similar to user i has a dimension of 1×K; uss is the user vector corresponding to the topK users, with a dimension of K×N, where N represents the total number of items; signif i ​​Represents the statistical results of the significant keyword aggregation of user i, with a dimension of 1×N; the '.' multiplication represents the multiplication of the corresponding elements of the vector; recScore i represents the recommended stock score of user i. b3. According to business requirements, just take the top M stocks.

[0059] An embodiment of the present invention also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute a personalized recommendation method based on a combination of user collaborative filtering and aggregate statistics.

[0060] Table 1 User score calculation parameters

[0061] category Corresponding score Weighted Weight User Positions Score1 W1 User selection Score2 W2 User's recent browsing history Score3 W3 ... ... ... other scorei wi

[0062] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0063] An electronic device provided by an embodiment of the present invention, such as Figure 2 As shown, the electronic device 800 includes a memory 801 and a processor 802. The memory stores a computer program that can be run on the processor. When the processor executes the computer program, the steps of the method provided in the above embodiment are implemented.

[0064] like Figure 2 As shown, the electronic device further includes: a bus 803 and a communication interface 804, a processor 802, a communication interface 804 and a memory 801 are connected via the bus 803; the processor 802 is used to execute executable modules stored in the memory 801, such as computer programs.

[0065] The memory 801 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 804 (which may be wired or wireless), and the Internet, wide area network, local area network or metropolitan area network may be used.

[0066] The bus 803 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0067] Among them, the memory 801 is used to store programs, and the processor 802 executes the program after receiving the execution instruction. The method executed by the device defined by the process disclosed in any embodiment of the present invention can be applied to the processor 802 or implemented by the processor 802.

[0068] The processor 802 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 802. The above processor 802 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801 and completes the steps of the above method in combination with its hardware.

[0069] Corresponding to the above method, an embodiment of the present invention further provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above method.

[0070] The CPU stress testing device and computer-readable storage medium of the server provided in the embodiments of the present invention have the same technical features as the CPU stress testing method provided in the above embodiments, so they can also solve the same technical problems and achieve the same technical effects.

[0071] The present invention can be extended to the stress test of storage devices such as hard disks and memory.

[0072] The device provided in the embodiment of the present invention may be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present invention are the same as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment. It can be clearly understood by those skilled in the art that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0073] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0074] For another example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some communication interface, device or unit, which can be electrical, mechanical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0076] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0077] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0078] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0079] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the product of the invention is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific position, be constructed and operated in a specific position, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0080] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0081] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics, characterized by: The recommendation system includes a data preprocessing module, a user similarity calculation module and a secondary sorting module based on significant aggregation statistics. The recommendation method specifically includes the following steps: S1. After the original user selected stocks and holding stocks data are processed by the data preprocessing module, a clean user stock data set and the corresponding user sparse vector are obtained; S2. According to the input user and user vector, the cosine similarity distance formula in the user similarity calculation module is used to calculate the user similarity and find the top similar users; S3, according to the topk similar users obtained in step S2, perform secondary sorting through a secondary sorting module based on significant aggregation statistics, and return the topM item sorting results; The secondary sorting method in step S3 specifically includes the following steps: b1. Perform significant aggregation on the topK users, then count the stocks purchased by the topK users, and then divide each statistical result by the overall global statistical result to obtain the score of each statistical stock. The formula is as follows: Among them, card pre (stock l ) means stock l The number of occurrences of card in the topK user data set in the previous section; total (stock l ) means stock l The number of occurrences in the overall data set of the backend; card total Indicates the total number of all stocks; card pre_total Represents the total number of stocks in the previous data set; b2. Calculate the recommended stock scores. The specific calculation formula is as follows: recScore i =cos<u i ,u * >·uss.signif i Among them, cos i ,u * > indicates that the similarity vector of the topk users most similar to user i has a dimension of 1×K; uss is the user vector corresponding to the topK users, with a dimension of K×N, where N represents the total number of items; signif i Represents the statistical results of the significant keyword aggregation of user i, with a dimension of 1×N; the '.' multiplication represents the multiplication of the corresponding elements of the vector; recScore i represents the recommended stock score of user i,​ b3. According to business requirements, select the top M stocks; The significant aggregation in step b1 is used to display unusual words of a specific group.

2. The personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics according to claim 1, characterized in that: The method for processing the original user selected stocks and holding stocks data by the data preprocessing module in step S1 specifically includes the following steps: a1. The weighted formula for the user's selected stocks and holdings data and subsequent new feature dimension scores is as follows: in In formula 1, uss jk represents the interest score of user j in stock k; score i Indicated in USS jk When calculating, the score of the i-th feature, the specific score i The value of is determined by business logic; i Represents the weighted weight of the i-th feature; a2. Pre-calculate the normalized eigenvector. From the vector cosine calculation formula, we know that From formula 2, we can know that the cosine distance calculation pre-calculates the normalized user vector, and the pre-calculation formula is:

3. The personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics according to claim 2 is characterized by: In step a1, w i It is used to weight the scores of each feature.

4. The personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics according to claim 2, characterized in that: In step S2, the corresponding user pre-calculated vector is found according to the input user id, and the user similarity score is calculated according to the vector cosine calculation formula to obtain the topK similar users. In this process, the speed is improved by the retrieval capability based on the common database. Since the sparse stock vectors selected by the user's holdings are not reduced, the results are controllable and interpretable.

5. The personalized recommendation method based on the combination of user collaborative filtering and aggregate statistics according to claim 4 is characterized by: The common database is one of elasticsearch and milvus.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 5.

Images