Information integration method, device, equipment, readable storage medium and program product

Abstract

The present invention discloses an information integration method, device, equipment, readable storage medium and program product, the method comprising: obtaining video data collected by shooting offline stores, performing human body tracking analysis on each frame image in the video data, and obtaining offline actual information corresponding to each customer, wherein the offline actual information includes customer feature information and time attribute information of the customer in the store; matching each customer with online order information according to the time attribute information to obtain an ordering customer, wherein the online order information is recorded during the shooting of the video data; and associating and storing the offline actual information corresponding to the ordering customer with the online order information. The present invention realizes the integration of offline actual information of offline store customers with online order information, provides more abundant marketing strategy reference information for offline store marketing personnel, and is conducive to the store to achieve precision marketing.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an information integration method, device, equipment, readable storage medium and program product. Background Art

[0002] With the development of science and technology, my country's retail industry has made great progress. Currently, online retail has become the main way to obtain customer information. However, driven by new retail, in order to achieve precision marketing, how offline stores obtain customer information has also become very important. However, offline stores can generally only record some simple sales data, and it is difficult to obtain more offline customer data related to customers. Therefore, at present, how to combine online and offline customer information has also become a major technical difficulty. Summary of the invention

[0003] The main purpose of the present invention is to provide an information integration method, device, equipment, readable storage medium and program product, aiming to solve the technical problem of how to combine online and offline customer information of offline stores.

[0004] To achieve the above object, the present invention provides an information integration method, which comprises the following steps:

[0005] Obtain video data collected by shooting offline stores, perform human body tracking analysis on each frame of the video data, and obtain offline actual information corresponding to each customer, wherein the offline actual information includes customer feature information and time attribute information of the customer in the store;

[0006] Matching each customer with online order information according to the time attribute information to obtain an ordering customer, wherein the online order information is recorded during the shooting of the video data;

[0007] The offline actual information and online order information corresponding to the ordering customer are associated and stored.

[0008] Optionally, the step of performing human body tracking analysis on each frame image in the video data to obtain offline actual information corresponding to each customer includes:

[0009] Performing human body region detection on each frame image in the video data to obtain the human body region in each frame image;

[0010] Extracting features from the human body regions to obtain feature information corresponding to each of the human body regions;

[0011] Matching the human body regions in each frame image according to the feature information to obtain each human body region group, wherein one human body region group corresponds to one customer, and one human body region group includes at least two human body regions in different frame images that are successfully matched;

[0012] The characteristic information corresponding to the human body region group is used as the customer characteristic information of the corresponding customer, and the time attribute information of the corresponding customer in the store is obtained according to the frame image statistics of each human body region in the human body region group.

[0013] Optionally, the step of obtaining time attribute information of the corresponding customer in the store according to the frame image statistics of each human body region in the human body region group includes:

[0014] Determine the actual location of each human body region in the human body region group in the store according to the location of each human body region in the frame image;

[0015] Dividing the human body regions in the same actual location in the human body region group into the same group to obtain human body region groups;

[0016] The time stamp of the frame image of each human body region in the human body region group is associated with the actual location corresponding to the human body region group to obtain time attribute information of the customer corresponding to the human body region group at different actual locations in the store.

[0017] Optionally, the step of matching each customer with online order information according to the time attribute information to obtain the ordering customer includes:

[0018] Matching the stay time at the store checkout counter in the time attribute information with the order generation time in each online order information;

[0019] If the match is successful, it is determined that the customer corresponding to the time attribute information is the ordering customer.

[0020] Optionally, before the step of associating and storing the offline actual information corresponding to the ordering customer and the online order information, the method further includes:

[0021] Detecting whether the ordering customer is a customer who places an order through an online ordering system according to the order number in the online order information corresponding to the ordering customer;

[0022] If so, obtaining the historical order information of the ordering customer in the online ordering system, and associating and storing the offline actual information, online ordering information and the historical order information corresponding to the ordering customer;

[0023] If not, the step of associating and storing the offline actual information and online order information corresponding to the ordering customer is performed.

[0024] Optionally, after the step of acquiring video data collected by shooting offline stores, performing human body tracking analysis on each frame image in the video data, and obtaining offline actual information corresponding to each customer, the step further includes:

[0025] Match the customer characteristic information of each customer with the customer characteristic information of historical customers in a preset database to determine the customer type of each customer;

[0026] The step of associating and storing the offline actual information and online order information corresponding to the ordering customer comprises:

[0027] For an ordering customer whose customer type is a new customer, the offline actual information and online order information corresponding to the ordering customer are associated and stored in the preset database;

[0028] For non-ordering customers whose customer type is a new customer, the offline actual information corresponding to the non-ordering customer is marked with a non-ordering label and then stored in the preset database;

[0029] For an ordering customer whose customer type is an old customer, the online order information corresponding to the ordering customer is associated and stored with the historical information of the ordering customer in the preset database;

[0030] For non-ordering customers whose customer type is old customers, a non-ordering record is generated according to the actual offline information of the non-ordering customer, and the non-ordering record is associated with the historical information of the non-ordering customer in the preset database and stored.

[0031] Optionally, after the step of matching the customer characteristic information of each customer with the customer characteristic information of historical customers in a preset database to determine the customer type of each customer, the step further includes:

[0032] For a customer whose customer type is an old customer, historical order information of the old customer is obtained from the preset database, and recommended commodities for the old customer are determined according to the historical order information.

[0033] To achieve the above object, the present invention further provides an information integration device, the device comprising:

[0034] An analysis module is used to obtain video data collected by shooting offline stores, perform human body tracking analysis on each frame of the video data, and obtain offline actual information corresponding to each customer, wherein the offline actual information includes customer feature information and time attribute information of the customer in the store;

[0035] A matching module, used for matching each customer with online order information according to the time attribute information to obtain an ordering customer, wherein the online order information is recorded during the shooting of the video data;

[0036] The association module is used to associate and store the offline actual information and online order information corresponding to the ordering customer.

[0037] To achieve the above objectives, the present invention also provides an information integration device, which includes: a memory, a processor, and an information integration program stored in the memory and executable on the processor, wherein the information integration program implements the steps of the information integration method described above when executed by the processor.

[0038] In addition, to achieve the above-mentioned purpose, the present invention further proposes a computer-readable storage medium, on which an information integration program is stored, and when the information integration program is executed by a processor, the steps of the above-mentioned information integration method are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also proposes a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned information integration method are implemented.

[0040] In the present invention, by acquiring the video data collected by shooting offline stores and performing human body tracking analysis on each frame of the video data, the offline actual information of each customer is obtained, and the offline actual information includes the customer's characteristic information and the time attribute information in the store; according to the time attribute information, each customer is matched with the online order information recorded during the video data shooting to obtain the ordering customer, and the online order information and offline actual information corresponding to the ordering customer are associated and stored to complete the integration of online and offline information. In this way, the online order information that was originally unrelated to the customer is connected to the customer's offline actual information, and the offline actual information includes the customer's characteristic information, so that the marketing personnel of the offline store can classify the online order information according to the customer's characteristic information, so as to analyze the order data from the customer's characteristic dimension, thereby providing a strong reference for the marketing strategy of the offline store. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0042] Figure 2 It is a flowchart of the first embodiment of the information integration method of the present invention;

[0043] Figure 3 It is a schematic diagram of the functional modules of a preferred embodiment of the information integration device of the present invention.

[0044] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0046] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0047] It should be noted that the information integration device in the embodiment of the present invention can be a smart phone, a personal computer, a server and other devices, and no specific limitation is made here.

[0048] like Figure 1 As shown, the information integration device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0049] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the information integration device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0050] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an information integration program. The operating system is a program that manages and controls the hardware and software resources of the device and supports the operation of the information integration program and other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; and the processor 1001 can be used to call the information integration program stored in the memory 1005 and perform the following operations:

[0051] Obtain video data collected by shooting offline stores, perform human body tracking analysis on each frame of the video data, and obtain offline actual information corresponding to each customer, wherein the offline actual information includes customer feature information and time attribute information of the customer in the store;

[0052] Matching each customer with online order information according to the time attribute information to obtain an ordering customer, wherein the online order information is recorded during the shooting of the video data;

[0053] The offline actual information and online order information corresponding to the ordering customer are associated and stored.

[0054] Furthermore, the step of performing human body tracking analysis on each frame image in the video data to obtain offline actual information corresponding to each customer includes:

[0055] Performing human body region detection on each frame image in the video data to obtain the human body region in each frame image;

[0056] Extracting features from the human body regions to obtain feature information corresponding to each of the human body regions;

[0057] Matching the human body regions in each frame image according to the feature information to obtain each human body region group, wherein one human body region group corresponds to one customer, and one human body region group includes at least two human body regions in different frame images that are successfully matched;

[0058] The characteristic information corresponding to the human body region group is used as the customer characteristic information of the corresponding customer, and the time attribute information of the corresponding customer in the store is obtained according to the frame image statistics of each human body region in the human body region group.

[0059] Furthermore, the step of obtaining time attribute information of the corresponding customer in the store according to the frame image statistics of each human body region in the human body region group includes:

[0060] Determine the actual location of each human body region in the human body region group in the store according to the location of each human body region in the frame image;

[0061] Dividing the human body regions in the same actual location in the human body region group into the same group to obtain human body region groups;

[0062] The time stamp of the frame image of each human body region in the human body region group is associated with the actual location corresponding to the human body region group to obtain time attribute information of the customer corresponding to the human body region group at different actual locations in the store.

[0063] Furthermore, the step of matching each customer with the online order information according to the time attribute information to obtain the ordering customer includes:

[0064] Matching the stay time at the store checkout counter in the time attribute information with the order generation time in each online order information;

[0065] If the match is successful, it is determined that the customer corresponding to the time attribute information is the ordering customer.

[0066] Furthermore, before the step of associating and storing the offline actual information and online order information corresponding to the ordering customer, the processor 1001 may also be used to call the information integration program stored in the memory 1005 and perform the following operations:

[0067] Detecting whether the ordering customer is a customer who places an order through an online ordering system according to the order number in the online order information corresponding to the ordering customer;

[0068] If so, obtaining the historical order information of the ordering customer in the online ordering system, and associating and storing the offline actual information, online ordering information and the historical order information corresponding to the ordering customer;

[0069] If not, the step of associating and storing the offline actual information and online order information corresponding to the ordering customer is performed.

[0070] Further, after the step of acquiring the video data collected by shooting offline stores, performing human body tracking analysis on each frame of the video data, and obtaining offline actual information corresponding to each customer, the processor 1001 can also be used to call the information integration program stored in the memory 1005, and perform the following operations:

[0071] Match the customer characteristic information of each customer with the customer characteristic information of historical customers in a preset database to determine the customer type of each customer;

[0072] The step of associating and storing the offline actual information and online order information corresponding to the ordering customer comprises:

[0073] For an ordering customer whose customer type is a new customer, the offline actual information and online order information corresponding to the ordering customer are associated and stored in the preset database;

[0074] For non-ordering customers whose customer type is a new customer, the offline actual information corresponding to the non-ordering customer is marked with a non-ordering label and then stored in the preset database;

[0075] For an ordering customer whose customer type is an old customer, the online order information corresponding to the ordering customer is associated and stored with the historical information of the ordering customer in the preset database;

[0076] For non-ordering customers whose customer type is old customers, a non-ordering record is generated according to the actual offline information of the non-ordering customer, and the non-ordering record is associated with the historical information of the non-ordering customer in the preset database and stored.

[0077] Furthermore, after the step of matching the customer characteristic information of each customer with the customer characteristic information of historical customers in the preset database to determine the customer type of each customer, the processor 1001 may also be used to call the information integration program stored in the memory 1005 and perform the following operations:

[0078] For a customer whose customer type is an old customer, historical order information of the old customer is obtained from the preset database, and recommended commodities for the old customer are determined according to the historical order information.

[0079] Based on the above structure, various embodiments of the information integration method are proposed.

[0080] Reference Figure 2 , Figure 2 It is a flowchart of the first embodiment of the information integration method of the present invention.

[0081] The embodiments of the present invention provide embodiments of the information integration method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here. The execution subjects of each embodiment of the information integration method of the present invention may be devices such as smart phones, personal computers, and servers. For ease of description, the execution subjects are omitted in the following embodiments. In this embodiment, the information integration method includes:

[0082] Step S10, obtaining video data collected by shooting offline stores, performing human body tracking analysis on each frame of the video data, and obtaining offline actual information corresponding to each customer, wherein the offline actual information includes customer feature information and time attribute information of the customer in the store;

[0083] Existing offline stores can only record the actual sales data of the stores, that is, they can only record the amount of each order (order) and the type of goods and other customer-irrelevant information (hereinafter referred to as online order information), but cannot obtain the actual offline information of the ordering customers, such as the customer's age, gender, appearance and other characteristic information, which is not conducive to the realization of precision marketing. Therefore, this embodiment proposes a method based on computer vision to obtain the customer's offline actual information, and integrate the customer's online order information, to achieve online and offline information integration, and then provide more and better reference data for precision marketing.

[0084] In this embodiment, a camera can be installed at a suitable location in an offline store to capture images or videos of the store's interior. The store's video data can be acquired from the camera in real time, or when it is necessary to integrate online and offline information of customers in the store, the video data captured within a period of time can be acquired from the camera, for example, once every hour.

[0085] The video data includes multiple consecutive frames of images, and human tracking analysis can be performed on all the frames of the video data or on frames extracted at equal intervals. Human tracking analysis can be achieved using existing commonly used human tracking technology, the purpose of which is to distinguish each customer appearing in the video according to their human features, obtain the position of each customer in each frame of the image, and determine the position of each customer in the store based on the position in the image, and then determine the stay time of each customer at each actual location in the store.

[0086] Through human body tracking analysis, the actual offline information corresponding to each customer appearing in the video data can be obtained; the actual offline information may include customer feature information, which may be the customer's facial feature information, and may also include feature information such as the customer's gender and age; the actual offline information also includes the customer's time attribute information in the store. When only information integration of ordering customers is required, the time attribute information may be the customer's stay time at the checkout counter in the store, and the stay time is generally a period of time.

[0087] Step S20, matching each customer with online order information according to the time attribute information to obtain an ordering customer, wherein the online order information is recorded during the shooting of the video data;

[0088] During the video shooting, the cash register device at the cash register in the store recorded the customer's order information, which may include basic information about the type of goods and the amount of the goods, as well as other additional information such as whether an invoice is issued. In addition, the order information also corresponds to an order generation time.

[0089] If the offline store also supports ordering through an online ordering system, for example, ordering through an APP, then when the customer places an order through the online ordering system, the online ordering system will generate an order information. When the customer takes the goods corresponding to the order from the offline store, the store cashier can enter the order number of the order in the cash register. The cash register can record the timestamp when the input operation is completed as the completion time of the order corresponding to the order number, and associate it with the order number.

[0090] When integrating online and offline information, the order information recorded during the shooting time period of the video data can be pulled from the cash register device, and the order number recorded during the time period through the online ordering system can be pulled, and the order information corresponding to the order number can be pulled from the online ordering system. The pulled order information is collectively referred to as online order information.

[0091] After obtaining the actual offline information of each customer by analyzing the video data, each customer can be matched with the online order information according to the customer's time attribute information to determine whether each customer is an ordering customer of the online order. If the customer successfully matches the online order information, it means that the customer is the ordering customer of the online order information. If the customer does not match any online order information, it means that the customer is not an ordering customer, that is, a non-ordering customer.

[0092] The step S20 comprises:

[0093] Step S201, matching the stay time at the store checkout counter in the time attribute information with the order generation time in each online order information;

[0094] Step S202: If the match is successful, it is determined that the customer corresponding to the time attribute information is the ordering customer.

[0095] Specifically, the customer's time attribute information includes the customer's stay time at the checkout counter in the store. This stay time is matched with the order generation time corresponding to the online order information. If the match is successful, it is determined that the customer is the ordering customer of the online order information. If the match fails, it is determined that the customer is not the ordering customer of the online order information.

[0096] Furthermore, if the online order information is pulled from an online ordering system, the customer's stay time at the checkout counter in the store in the time attribute information is matched with the completion time corresponding to the online order information. If the match is successful, it is determined that the customer is the ordering customer of the online order information. If the match fails, it is determined that the customer is not the ordering customer of the online order information.

[0097] The order generation time is generally a time point. If the time point falls within the time period of the stay time, it can be determined that the order generation time matches the stay time. If not, it is determined that they do not match. The same is true for matching the stay time with the order completion time.

[0098] Step S30, the offline actual information and online order information corresponding to the ordering customer are associated and stored.

[0099] After determining the ordering customers among the various customers and the online order information corresponding to each ordering customer, the offline actual information corresponding to the ordering customer and the online order information may be associated and stored.

[0100] In addition, for non-ordering customers among each customer, the actual offline information of the non-ordering customers can also be stored, so that marketing personnel can analyze the order rate of offline stores based on the stored information and adjust the marketing strategy based on the order rate.

[0101] In this embodiment, by acquiring the video data collected by shooting offline stores and performing human body tracking analysis on each frame of the video data, the offline actual information of each customer is obtained, and the offline actual information includes the customer's characteristic information and the time attribute information in the store; according to the time attribute information, each customer is matched with the online order information recorded during the video data shooting to obtain the ordering customer, and the online order information and offline actual information corresponding to the ordering customer are associated and stored to complete the integration of online and offline information. In this way, the online order information that was originally unrelated to the customer is connected to the customer's offline actual information, and the offline actual information includes the customer's characteristic information, so that the marketing personnel of the offline store can classify the online order information according to the customer's characteristic information, so as to analyze the order data from the customer's characteristic dimension, thereby providing a strong reference for the marketing strategy of the offline store, which is conducive to the store to achieve precision marketing.

[0102] Furthermore, online and offline information can be associated and stored in a preset database. When an analysis instruction is received, the order information in the database is analyzed and the analysis results are output visually. Specifically, the analysis instruction may carry a standard for dividing the order information, for example, by gender, by age, by customer, etc.; then, when the analysis instruction is received, the order information in the database is divided into categories according to the division standard in the analysis instruction, such as dividing the orders in the database into orders for male customers and orders for female customers. And the division results are output visually, for example, in the form of a bar chart, a table, a line chart, etc., so that the marketing personnel of the offline store can formulate marketing strategies with reference to the output data. Compared with formulating marketing strategies based solely on order information, this embodiment provides richer and more accurate reference data, thereby improving the accuracy of the marketing strategy.

[0103] Further, based on the above first embodiment, a second embodiment of the information integration method of the present invention is proposed. In this embodiment, the step of performing human body tracking analysis on each frame image in the video data in step S10 to obtain offline actual information corresponding to each customer includes:

[0104] Step S101, performing human body region detection on each frame image in the video data to obtain the human body region in each frame image;

[0105] Furthermore, in this embodiment, a method for performing human body tracking analysis on video data is provided. Specifically, human body region detection is performed on each frame image in the video data to obtain the human body region in each frame image. The human body region may refer to a region including only a face region or a region including the entire human body. The method for performing human body region detection on a frame image may be to use the entire region of a face or a human body as a target, and to detect the frame image using an existing target detection method to obtain the human body region in the frame image. It should be noted that multiple human body regions may be detected in a frame image, that is, multiple customers appear in a frame image at the same time.

[0106] Step S102, extracting features from the human body regions to obtain feature information corresponding to each of the human body regions;

[0107] After obtaining each human body region in each frame image, feature extraction is performed on each human body region. Specifically, when the human body region includes a face region, facial attribute feature extraction can be performed on the face region, which can be implemented by using an existing image feature extraction model or algorithm to extract facial feature information, which can be in the form of a feature vector, or can also include feature information such as gender and age. When the human body region is the entire human body region, in addition to extracting facial attribute features from the face region in the region to obtain facial feature information, external attribute feature extraction can also be performed on the part of the region other than the face region, such as by using an existing commonly used semantic segmentation model or algorithm, that is, glasses, hats, and clothing are used as categories for semantic segmentation, and external attribute feature information such as whether glasses are worn, whether hats are worn, and clothing styles are identified.

[0108] After feature extraction, the feature information corresponding to each human body area can be obtained.

[0109] Step S103, matching the human body regions in each frame image according to the feature information to obtain each human body region group, wherein one human body region group corresponds to one customer, and one human body region group includes at least two human body regions in different frame images that are successfully matched;

[0110] According to the characteristic information, the human body regions in each frame image are matched to obtain various human body region groups, that is, the human body regions that are successfully matched are divided into the same group. It should be noted that in one frame image, one human body region corresponds to one customer, and it is impossible for one customer to correspond to two human body regions in one frame image, so the human body regions in different frame images are matched. Taking two frames of images as an example, a human body region in one frame image is matched with each human body region in the other frame image respectively. If the match with one of the human body regions is successful, the human body region and the human body region that is successfully matched are divided into the same human body region group, that is, the two human body regions correspond to the same customer; such a match is performed for each human body region in the frame image, that is, the matching process of the two frames of images is completed. The same is true for multiple frames of images. After matching, multiple human body region groups are obtained, one human body region group corresponds to one customer, and one human body region group includes at least two human body regions, that is, one human body region includes multiple human body regions, and the frame images in which the multiple human body regions are located are different; then one human body region group corresponds to multiple frames of images, and these multiple frames of images are generally continuous.

[0111] Specifically, matching two human body regions specifically means matching the feature information of the two human body regions. For example, when the feature information includes facial attribute feature information and is in the form of a feature vector, the distance between the two feature vectors, such as the Euclidean distance, can be calculated. If the distance is less than a certain distance, the two human body regions are determined to match. If not, they are determined not to match. To improve matching efficiency, when the facial attribute feature information also includes the gender and age of the customer, the gender and age of the two human body regions can be compared first. If they are not the same, it is directly determined that the two human body regions do not match. If they are the same, the feature vectors are compared again.

[0112] Step S104, taking the characteristic information corresponding to the human body region group as the customer characteristic information of the corresponding customer, and obtaining the time attribute information of the corresponding customer in the store according to the frame image statistics of each human body region in the human body region group.

[0113] For each human body region group, the characteristic information corresponding to the human body region group is used as the customer characteristic information of the customer corresponding to the human body region group. Specifically, each human body region corresponds to a characteristic information, and the characteristic information corresponding to one of the human body regions can be selected as the customer characteristic information of the customer corresponding to the human body region group. For each human body region group, the time attribute information of the customer corresponding to the human body region group in the store can be obtained based on the frame image where each human body region in the human body region group is located. That is, a human body region group corresponds to multiple frames of images, and each frame of image corresponds to a timestamp, so that the time the customer corresponding to the human body region group stays in the store can be obtained based on the timestamp of the image. Specifically, based on the position of the human body region in the image, the customer's stay time at the checkout counter can be obtained, and the stay time can be used as the customer's time attribute information.

[0114] Furthermore, the step S101 includes:

[0115] Step S1011, determining the actual location of each human body region in the human body region group in the store according to the location of each human body region in the frame image;

[0116] Specifically, in one embodiment, for each human body region group, the actual location of each human body region in the store can be determined based on the position of each human body region in the human body region group in the frame image. Specifically, the area in the store can be divided in advance, such as dividing a certain range around the cash register into a cash register area, and dividing a certain range around the merchandise shelf into a merchandise shelf area, and the actual location is these areas. The actual location corresponding to each pixel point in the frame image is pre-calibrated, then, the actual location of the human body region can be determined based on the pixel point coordinates of the human body region in the image, for example, it can be determined whether the human body region is in the cash register area, the merchandise shelf area, the dining table area, etc.

[0117] Step S1012, dividing the human body regions in the same actual location in the human body region group into the same group to obtain human body region groups;

[0118] After obtaining the actual location corresponding to each human body area in the human body area group, each human body area can be divided according to the actual location, that is, the human body areas at the same actual location are divided into the same human body area group. After the division, the human body area group corresponding to each actual location can be obtained.

[0119] For example, if the human body regions in the cashier area are divided into a group, the frame images corresponding to the human body regions in this group are the frame images taken when the customers corresponding to the human body region group are in the cashier area. If the human body regions in the merchandise shelf area are divided into a group, the frame images corresponding to the human body regions in this group are the frame images taken when the customers corresponding to the human body region group are in the merchandise shelf area.

[0120] Step S1013, associating the time stamp of the frame image of each human body region in the human body region group with the actual location corresponding to the human body region group, and obtaining time attribute information of the customer corresponding to the human body region group at different actual locations in the store.

[0121] By associating the timestamps of the frame images of each human region in the human region group with the actual location corresponding to the human region group, the time attribute information of the customer at the actual location corresponding to the human region group can be obtained, that is, the time attribute information includes the actual location of the customer and the time the customer spends at the actual location. Here, the span of the timestamps of each frame image can be used as the stay time, so the time attribute information can be the stay time of the customer at the actual location. For example, if the actual location corresponding to the human region group is the cashier area, the time attribute information of the customer includes the stay time of the customer at the cashier area.

[0122] The actual location corresponding to each human body region group is different, so the time attribute information of the customer corresponding to the human body region group at different actual locations can be obtained.

[0123] In this embodiment, by performing human body tracking analysis on each frame of image in the video data, the offline actual information corresponding to each customer is obtained. The offline actual information includes time attribute information of different actual locations of each customer in the store. By integrating the customer's offline actual information with the online order information, more abundant reference information can be provided for the store's marketing personnel. For example, by analyzing the residence time of each customer at each actual location in the store, the customer's preference for the layout of each area in the store can be determined, and then a more accurate store layout adjustment strategy can be made based on the analysis results to improve the store's customer favorability.

[0124] Further, based on the above-mentioned first and / or second embodiments, a third embodiment of the information integration method of the present invention is proposed. In this embodiment, before step S30, the method further includes:

[0125] Step S40, detecting whether the ordering customer is a customer who places an order through an online ordering system according to the order number in the online order information corresponding to the ordering customer;

[0126] Furthermore, in this embodiment, if offline stores support customers to place orders in the online ordering system, then for customers appearing in the video data, if they place orders through the online ordering system, the customer's online and offline information can be associated and stored together with the historical order information of their online ordering system.

[0127] Specifically, after determining the ordering customer, it is possible to detect whether the ordering customer is a customer who places an order through the online ordering system based on the order number in the online order information corresponding to the ordering customer, that is, to determine whether the ordering customer places the order through the online ordering system. Among them, for orders placed by customers through the online ordering system, a different order numbering rule can be used from orders placed through the cash register device, so it is possible to distinguish whether the online order information is generated in the online ordering system based on the order number of the online order information.

[0128] Step S50: If yes, then obtain the historical order information of the ordering customer in the online ordering system, and associate and store the offline actual information, online ordering information and historical order information corresponding to the ordering customer;

[0129] If it is determined that the ordering customer is a customer who places an order through the online ordering system, the historical order information of the ordering customer in the online ordering system can be obtained from the online ordering system. Specifically, according to the order number of the online order information corresponding to the ordering customer, the order information of the customer who places an order with the same order number in the historical record can be pulled out from the online ordering system. It should be noted that each order information in the online ordering system will have corresponding customer information, such as the customer's account information, phone number, etc. in the online ordering system.

[0130] The acquired historical order information of the ordering customer is associated with the offline actual information and online order information of the ordering customer and stored.

[0131] If not, execute step S30.

[0132] If it is determined that the ordering customer is not a customer who places an order through the online ordering system, the actual offline information of the ordering customer and the online ordering information may be associated and stored.

[0133] In this embodiment, by identifying the ordering method of the ordering customer, when the ordering customer places an order through an online ordering system, the ordering customer's actual offline information is associated and stored with the current online order information, and the ordering customer's historical order information in the online ordering system is also associated and stored, thereby achieving more accurate online and offline information integration, thereby providing richer and more accurate reference information for precision marketing.

[0134] Further, based on the first, second and / or third embodiments above, a fourth embodiment of the information integration method of the present invention is proposed. In this embodiment, after step S10, the method further includes:

[0135] Step S60, matching the customer characteristic information of each customer with the customer characteristic information of historical customers in a preset database to determine the customer type of each customer;

[0136] Furthermore, in this embodiment, online and offline information may be associated and stored in a preset database, and the database then stores online order information and offline actual information of the store's historical customers.

[0137] After obtaining the offline actual information of each customer by performing human body tracking analysis on the video data, the customer feature information in the offline actual information of the customer can be matched with the customer feature information of the historical customers in the database to determine the customer type of the customer, wherein the customer type includes new customers and old customers. The method for matching the customer feature information can be the same as the method for matching the customer feature information in the above embodiment, which will not be described in detail here.

[0138] If the match is successful, the customer is determined to be an old customer. If it does not match any historical customer in the database, the customer is determined to be a new customer. In other words, if the customer appearing in the video is a customer that has been recorded before, the customer is an old customer. If the customer appearing in the video is a customer that has not been recorded before, the customer is a new customer.

[0139] The step S30 comprises:

[0140] Step S301, for an ordering customer whose customer type is a new customer, the offline actual information and online order information corresponding to the ordering customer are associated and stored in the preset database;

[0141] Then, after determining the ordering customers among each customer, each customer appearing in the video data can be divided into four categories: ordering customers whose customer type is new customers, non-ordering customers whose customer type is new customers, ordering customers whose customer type is old customers, and non-ordering customers whose customer type is old customers.

[0142] For an ordering customer whose customer type is a new customer, the offline actual information and online order information corresponding to the ordering customer can be associated and stored in the database. Because it is a new customer, there is no historical information of the customer in the database, so a file is created for the customer in the database and the first record is stored.

[0143] Step S302, for non-ordering customers whose customer type is new customers, the offline actual information corresponding to the non-ordering customers is marked with a non-ordering label and then stored in the preset database;

[0144] For non-ordering customers whose customer type is new customers, since the customer is a new customer and has not placed an order, a file can be created for the customer in the database, and the customer's actual offline information can be stored as a record, and the offline actual information can be labeled as not having placed an order to distinguish it from ordering customers.

[0145] Step S303, for an ordering customer whose customer type is an old customer, the online order information corresponding to the ordering customer is associated and stored with the historical information of the ordering customer in the preset database;

[0146] For customers who are old customers placing orders, the database records the customer's historical information, which includes the customer's actual offline information and historical order information. Therefore, the customer's online order information can be associated with the historical information and stored, and the actual offline information may not be stored to avoid duplication.

[0147] Step S304, for a non-ordering customer whose customer type is an old customer, a non-ordering record is generated according to the actual offline information of the non-ordering customer, and the non-ordering record is associated with the historical information of the non-ordering customer in the preset database and stored.

[0148] For non-ordering customers who are old customers, the database records the customer's historical information, which includes the customer's actual offline information and historical order information. Although the customer entered the store this time, he did not place an order, so a non-order record can be generated and stored in association with the customer's historical information. The actual offline information may not be stored to avoid duplication.

[0149] In this embodiment, by matching the customer's characteristic information with the characteristic information of historical customers in the database, the customer type can be determined, thereby facilitating accurate classification of customers. In addition, the actual offline information of non-ordering customers is also recorded, so that marketing personnel can formulate marketing strategies based on more actual offline information of customers, for example, by analyzing the ratio of non-ordering customers to ordering customers to adjust the store's marketing strategy.

[0150] Furthermore, after step S60, the method further includes:

[0151] Step S70, for a customer whose customer type is an old customer, obtaining the historical order information of the old customer from the preset database, and determining the recommended products for the old customer according to the historical order information.

[0152] Furthermore, after analyzing and determining the customer type of each customer, for customers whose customer type is an old customer, the historical order information of the old customer can be obtained from the preset database, and the recommended products for the old customer can be determined based on the historical order information. For example, the product that the customer ordered the most times can be counted from the historical order information and used as the recommended product.

[0153] Furthermore, when the video data is analyzed to determine that the old customer is in the checkout area, the image of the recommended product can be output to the display screen at the checkout counter, so that the customer can place an order based on the recommended products on the display screen, thereby increasing the customer's favorability towards the store and improving ordering efficiency.

[0154] In addition, the present invention also provides an information integration device, referring to Figure 3 , the device comprises:

[0155] The analysis module 10 is used to obtain the video data collected by shooting offline stores, perform human body tracking analysis on each frame image in the video data, and obtain offline actual information corresponding to each customer, wherein the offline actual information includes customer feature information and time attribute information of the customer in the store;

[0156] A matching module 20, configured to match each customer with online order information according to the time attribute information to obtain an ordering customer, wherein the online order information is recorded during the shooting of the video data;

[0157] The association module 30 is used to associate and store the offline actual information and online order information corresponding to the ordering customer.

[0158] Furthermore, the analysis module 10 includes:

[0159] A detection unit, used to perform human body region detection on each frame image in the video data to obtain the human body region in each frame image;

[0160] An extraction unit, used for extracting features from the human body regions to obtain feature information corresponding to each of the human body regions;

[0161] A first matching unit is used to match the human body regions in each frame image according to the feature information to obtain each human body region group, wherein one human body region group corresponds to one customer, and one human body region group includes at least two human body regions in different frame images that are successfully matched;

[0162] The first determination unit is used to use the characteristic information corresponding to the human body region group as the customer characteristic information of the corresponding customer, and obtain the time attribute information of the corresponding customer in the store according to the frame image statistics of each human body region in the human body region group.

[0163] Further, the first determining unit includes:

[0164] A determination subunit, used to determine the actual location of each human body region in the human body region group in the store according to the location of each human body region in the frame image;

[0165] A division subunit, used for dividing the human body regions in the same actual location in the human body region group into the same group to obtain human body region groups;

[0166] The association subunit is used to associate the time stamp of the frame image of each human body region in the human body region group with the actual location corresponding to the human body region group to obtain the time attribute information of the customer corresponding to the human body region group at different actual locations in the store.

[0167] Further, the matching module 20 includes:

[0168] A second matching unit, configured to match the stay time at the store checkout counter in the time attribute information with the order generation time in each online order information;

[0169] The second determining unit is configured to determine that the customer corresponding to the time attribute information is an ordering customer if the match is successful.

[0170] Furthermore, the device also includes:

[0171] A detection module, used to detect whether the ordering customer is a customer who places an order through an online ordering system according to the order number in the online order information corresponding to the ordering customer;

[0172] The association module 30 is further used to: if yes, obtain the historical order information of the ordering customer in the online ordering system, and associate and store the offline actual information, online ordering information and the historical order information corresponding to the ordering customer;

[0173] If not, the offline actual information and online order information corresponding to the ordering customer are associated and stored.

[0174] Furthermore, the matching module 20 is used to match the customer characteristic information of each customer with the customer characteristic information of historical customers in a preset database to determine the customer type of each customer;

[0175] The association module is also used for:

[0176] For an ordering customer whose customer type is a new customer, the offline actual information and online order information corresponding to the ordering customer are associated and stored in the preset database;

[0177] For non-ordering customers whose customer type is a new customer, the offline actual information corresponding to the non-ordering customer is marked with a non-ordering label and then stored in the preset database;

[0178] For an ordering customer whose customer type is an old customer, the online order information corresponding to the ordering customer is associated and stored with the historical information of the ordering customer in the preset database;

[0179] For non-ordering customers whose customer type is old customers, a non-ordering record is generated according to the actual offline information of the non-ordering customer, and the non-ordering record is associated with the historical information of the non-ordering customer in the preset database and stored.

[0180] Furthermore, the device also includes:

[0181] The recommendation module is used to obtain the historical order information of the old customer from the preset database for the customer type being the old customer, and determine the recommended products for the old customer according to the historical order information.

[0182] The expanded contents of the specific implementation of the information integration device of the present invention are basically the same as the embodiments of the above-mentioned information integration method, and will not be elaborated here.

[0183] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which an information integration program is stored. When the information integration program is executed by a processor, the steps of the information integration method described below are implemented.

[0184] In addition, the present invention also provides a computer program product, including a computer program, which implements the steps of the above-mentioned information integration method when executed by a processor.

[0185] The various embodiments of the information integration device, computer-readable storage medium, and computer program product of the present invention may all refer to the various embodiments of the information integration method of the present invention, and will not be described in detail here.

[0186] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0187] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0188] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0189] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An information integration method, characterized in that: The method comprises the following steps: Obtain video data collected by shooting offline stores, perform human body tracking analysis on each frame of the video data, and obtain offline actual information corresponding to each customer, wherein the offline actual information includes customer feature information and time attribute information of the customer in the store; Matching each customer with online order information according to the time attribute information to obtain an ordering customer, wherein the online order information is recorded during the shooting of the video data; The offline actual information and online order information corresponding to the ordering customer are associated and stored; The step of matching each customer with the online order information according to the time attribute information to obtain the ordering customer includes: Matching the stay time at the store cashier counter in the time attribute information with the order generation time point in each online order information; If the order generation time point is within the stay period, it is determined that the customer corresponding to the time attribute information is the ordering customer.

2. The information integration method according to claim 1, characterized in that: The step of performing human body tracking analysis on each frame image in the video data to obtain offline actual information corresponding to each customer includes: Performing human body region detection on each frame image in the video data to obtain the human body region in each frame image; Extracting features from the human body regions to obtain feature information corresponding to each of the human body regions; Matching the human body regions in each frame image according to the feature information to obtain each human body region group, wherein one human body region group corresponds to one customer, and one human body region group includes at least two human body regions in different frame images that are successfully matched; The characteristic information corresponding to the human body region group is used as the customer characteristic information of the corresponding customer, and the time attribute information of the corresponding customer in the store is obtained according to the frame image statistics of each human body region in the human body region group.

3. The information integration method according to claim 2, characterized in that: The step of obtaining the time attribute information of the corresponding customer in the store according to the frame image statistics of each human body region in the human body region group comprises: Determine the actual location of each human body region in the human body region group in the store according to the location of each human body region in the frame image; Dividing the human body regions in the same actual location in the human body region group into the same group to obtain human body region groups; The time stamp of the frame image of each human body region in the human body region group is associated with the actual location corresponding to the human body region group to obtain time attribute information of the customer corresponding to the human body region group at different actual locations in the store.

4. The information integration method according to claim 1, characterized in that: Before the step of associating and storing the offline actual information and online order information corresponding to the ordering customer, the method further includes: Detecting whether the ordering customer is a customer who places an order through an online ordering system according to the order number in the online order information corresponding to the ordering customer; If so, obtaining the historical order information of the ordering customer in the online ordering system, and associating and storing the offline actual information, online ordering information and the historical order information corresponding to the ordering customer; If not, the step of associating and storing the offline actual information and online order information corresponding to the ordering customer is performed.

5. The information integration method according to any one of claims 1 to 3, characterized in that: After the step of acquiring the video data collected by shooting offline stores, performing human body tracking analysis on each frame image in the video data, and obtaining offline actual information corresponding to each customer, the method further includes: Match the customer characteristic information of each customer with the customer characteristic information of historical customers in a preset database to determine the customer type of each customer; The step of associating and storing the offline actual information and online order information corresponding to the ordering customer comprises: For an ordering customer whose customer type is a new customer, the offline actual information and online order information corresponding to the ordering customer are associated and stored in the preset database; For non-ordering customers whose customer type is a new customer, the offline actual information corresponding to the non-ordering customer is marked with a non-ordering label and then stored in the preset database; For an ordering customer whose customer type is an old customer, the online order information corresponding to the ordering customer is associated and stored with the historical information of the ordering customer in the preset database; For non-ordering customers whose customer type is old customers, a non-ordering record is generated according to the actual offline information of the non-ordering customer, and the non-ordering record is associated with the historical information of the non-ordering customer in the preset database and stored.

6. The information integration method according to claim 5, characterized in that: After the step of matching the customer characteristic information of each customer with the customer characteristic information of historical customers in the preset database to determine the customer type of each customer, the method further includes: For a customer whose customer type is an old customer, historical order information of the old customer is obtained from the preset database, and recommended commodities for the old customer are determined according to the historical order information.

7. An information integration device, characterized in that: The device comprises: An analysis module is used to obtain video data collected by shooting offline stores, perform human body tracking analysis on each frame of the video data, and obtain offline actual information corresponding to each customer, wherein the offline actual information includes customer feature information and time attribute information of the customer in the store; a matching module, configured to match each customer with online order information according to the time attribute information to obtain an ordering customer, wherein the online order information is recorded during the shooting of the video data; match the stay period at the store cashier counter in the time attribute information with the order generation time point in each online order information; if the order generation time point is within the stay period, determine that the customer corresponding to the time attribute information is the ordering customer; The association module is used to associate and store the offline actual information and online order information corresponding to the ordering customer.

8. An information integration device, characterized in that: The information integration device comprises: a memory, a processor, and an information integration program stored in the memory and executable on the processor. When the information integration program is executed by the processor, the steps of the information integration method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an information integration program, and when the information integration program is executed by the processor, the steps of the information integration method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the information integration method according to any one of claims 1 to 6 are implemented.

Images