Multi-modal data fusion-based shop patrol supervision method and system
Through multimodal data fusion and deep learning technology, combined with computer vision and natural language processing, the automation and intelligence of fast-selling product store inspection supervision has been realized, solving the high human resource consumption, low efficiency and data analysis difficulties in the existing technology. The problem of this issue improves store inspection efficiency and accuracy, reduces costs, and enhances corporate competitiveness and brand image.
Patent Information
- Application Number
- CN202410823102.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-25
- Publication Date
- 2024-12-17
AI Technical Summary
The existing fast-selling product store inspection methods rely on manual observation and manual recording, resulting in high human resource consumption, low efficiency, great subjectivity impact, and difficulty in data analysis. It cannot meet the advantages of multimodal data fusion based on single modal data analysis. , resulting in insufficient comprehensive and accurate feature representation, and insufficient real-time and applicability.
Using a multimodal data fusion method, combining deep learning and neural networks, image and voice data are collected through cameras, microphones and sound sensors, and features are extracted using computer vision and natural language processing technology to perform feature fusion, and multimodal Neural networks are jointly trained to realize problem detection and automated report generation.
It improves the efficiency and accuracy of store inspections, reduces human resource consumption and costs, enhances a comprehensive understanding of the store situation, achieves higher objectivity and adaptability, and improves the management level and brand image of store inspections.
Smart Images
Figure CN119150161A_ABST
Abstract
Description
Technical Field
[0001] The invention discloses a method and a system, and relates to the field of artificial intelligence technology, in particular to a store inspection supervision method and a system based on multimodal data fusion. Background Art
[0002] The existing fast-moving consumer goods store inspection and supervision methods mainly rely on manual observation and manual recording, which has some shortcomings, such as:
[0003] High human resource consumption: A large amount of human resources are required for store inspection and supervision, including dispatching store inspectors, training and management personnel, etc., which is costly;
[0004] Inefficiency: Manual observation and recording take a long time and are prone to omissions and errors, resulting in low store inspection efficiency;
[0005] Subjective influence: Since it relies on manual observation, the store inspection results are easily affected by the subjective factors of the observer, resulting in the results being not objective and accurate enough;
[0006] Difficulty in data analysis: The collected data is usually unstructured, making it difficult to conduct systematic analysis and mining, which limits the in-depth understanding and application of store inspection results. In order to solve these problems, some store inspection supervision methods based on artificial intelligence technology have emerged in recent years, but there are still some limitations:
[0007] Single-modal data analysis: focusing only on single-modal data, such as images or speech, while ignoring the advantages of multi-modal data fusion, resulting in incomplete and inaccurate feature representation;
[0008] Complexity of data processing: The processing and analysis of multimodal data requires expertise in multiple fields, such as computer vision and speech recognition. The technology is complex and imperfect.
[0009] Real-time and applicability: In actual applications, there are problems such as insufficient real-time performance and poor applicability, which cannot meet the rapidly changing market demands. Summary of the invention
[0010] In view of the problems of the prior art, the present invention provides a store inspection supervision method and system based on multimodal data fusion, which combines artificial intelligence technologies such as deep learning and neural networks to improve the efficiency, accuracy and intelligence level of store inspections.
[0011] The specific scheme proposed by the present invention is:
[0012] The present invention provides a store inspection supervision method based on multimodal data fusion, comprising:
[0013] Step 1: During the store inspection, multimodal data collection is performed on the shelf display, product placement, and customer flow in the store. The multimodal data includes image data and voice data.
[0014] Step 2: Process multimodal data: Combine computer vision technology and natural language processing technology to extract the features of image data, and combine speech recognition technology and natural language processing technology to extract the audio features and semantic features of speech data.
[0015] Step 3: Fuse the features of the image data, the audio features and the semantic features of the speech data to obtain the fused multimodal features. Use the multimodal neural network in the deep learning model to jointly train the recognition model based on the features of the image data, the audio features and the semantic features of the speech data to obtain the correlation information and mutual influence between the features.
[0016] Step 4: Use the recognition model to detect problems based on the fused multimodal features.
[0017] Step 5: Automatically generate a store inspection report based on the results of problem detection.
[0018] Furthermore, the multimodal data collection described in step 1 of the store inspection supervision method based on multimodal data fusion includes: deploying cameras, microphones and sound sensors for the in-store environment, and collecting in-store image data and voice data through the cameras, microphones and sound sensors respectively.
[0019] Furthermore, in the method for store inspection based on multimodal data fusion, the step 3 of fusing the features of the image data, the audio features and the semantic features of the speech data includes:
[0020] Using the feature fusion formula:
[0021]
[0022] Perform feature fusion, where F fused is the fused multimodal feature vector, F i represents a certain modal eigenvector, W i It corresponds to the weight of a modal feature vector and is used to adjust the contribution of each modal feature.
[0023] Furthermore, the generation of a store inspection report as described in step 5 of the store inspection supervision method based on multimodal data fusion includes: generating the classification, description and improvement suggestions of the problems in the store inspection report according to the results of problem detection, and adjusting and optimizing the recognition model according to the classification and description of the problems.
[0024] The present invention also provides a store inspection supervision system based on multimodal data fusion, including a data acquisition module, a data processing module, a problem detection module and a report generation module.
[0025] During the store inspection, the data collection module collects multimodal data on shelf display, product placement, and customer flow. The multimodal data includes image data and voice data.
[0026] The data processing module processes multimodal data: it combines computer vision technology and natural language processing technology to extract the features of image data, and combines speech recognition technology and natural language processing technology to extract the audio features and semantic features of speech data.
[0027] The features of image data, audio features and semantic features of speech data are fused to obtain fused multimodal features. The multimodal neural network in the deep learning model is used to jointly train the recognition model based on the features of image data, audio features and semantic features of speech data to obtain the correlation information and mutual influence between the features.
[0028] The problem detection module uses the recognition model to detect problems based on the fused multimodal features.
[0029] The report generation module automatically generates a store inspection report based on the results of problem detection.
[0030] Furthermore, the data acquisition module of the store inspection supervision system based on multimodal data fusion performs multimodal data acquisition, including: deploying cameras, microphones and sound sensors for the in-store environment, and collecting in-store image data and voice data through the cameras, microphones and sound sensors respectively.
[0031] Furthermore, the data processing module of the store inspection supervision system based on multimodal data fusion fuses the features of the image data, the audio features and the semantic features of the speech data, including:
[0032] Using the feature fusion formula:
[0033]
[0034] Perform feature fusion, where F fused is the fused multimodal feature vector, F i represents a certain modal eigenvector, W i It corresponds to the weight of a modal feature vector and is used to adjust the contribution of each modal feature.
[0035] Furthermore, the report generation module of the store inspection supervision system based on multimodal data fusion generates a store inspection report, including: generating the classification, description and improvement suggestions of the problem in the store inspection report according to the results of problem detection, and adjusting and optimizing the recognition model according to the classification and description of the problem.
[0036] The benefits of the present invention are:
[0037] Improve store inspection efficiency: Through automated data collection, processing and analysis, as well as intelligent problem detection and report generation, the store inspection process can be made more efficient, human resource consumption can be reduced, and time costs can be saved.
[0038] Improve store inspection accuracy: Combine multimodal data fusion technology, comprehensively consider multiple information sources such as images and voice, improve comprehensive understanding and accurate analysis of store conditions, reduce the impact of human factors, and improve the objectivity and accuracy of store inspection results.
[0039] Reduce store inspection costs: Reduce manual operation and management costs, improve store inspection efficiency and accuracy, thereby reducing the overall cost of store inspections and improving the competitiveness and profitability of the company.
[0040] Improve the intelligence level of store inspections: Use artificial intelligence technology to realize the automation and intelligence of the store inspection process, continuously optimize store inspection methods and techniques to enable them to adapt to changes in market demand, and improve store inspection management level and brand image. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a schematic diagram of the application process of the method of the present invention. DETAILED DESCRIPTION
[0042] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0043] Example 1
[0044] The present invention provides a store inspection supervision method based on multimodal data fusion, comprising:
[0045] Step 1: During the store inspection, multimodal data is collected on the shelf display, product placement, and customer flow in the store. The multimodal data includes image data and voice data.
[0046] The multimodal data collection may further include: deploying cameras, microphones and sound sensors for the in-store environment, and collecting in-store image data and voice data through the cameras, microphones and sound sensors respectively.
[0047] Step 2: Process multimodal data: Combine computer vision technology and natural language processing technology to extract features of image data, and combine speech recognition technology and natural language processing technology to extract audio features and semantic features of speech data. Analyze and understand the situation in the store.
[0048] Step 3: Fuse the features of the image data, the audio features and the semantic features of the speech data to obtain the fused multimodal features. Use the multimodal neural network in the deep learning model to jointly train the recognition model based on the features of the image data, the audio features and the semantic features of the speech data to obtain the correlation information and mutual influence between the features.
[0049] The step 3 described in which the features of the image data, the audio features of the speech data and the semantic features are integrated includes:
[0050] Using the feature fusion formula:
[0051]
[0052] Perform feature fusion, where F fused is the fused multimodal feature vector, F i represents a certain modal eigenvector, W i It represents the weight corresponding to a certain modal feature vector, which is used to adjust the contribution of each modal feature.
[0053] Step 4: Use the recognition model to detect problems based on the fused multimodal features. The trained model is used for problem detection. By analyzing multimodal data, possible problems in the store can be detected, such as out-of-stock shelves and irregular product placement.
[0054] Step 5: Automatically generate a store inspection report based on the problem detection results. The problem classification, description and improvement suggestions can be further generated in the store inspection report based on the problem detection results, and the recognition model can be adjusted and optimized based on the problem classification and description.
[0055] The implementation method of the method of the present invention involves multiple links such as equipment preparation, data collection and processing, feature fusion, problem detection, report generation, feedback and optimization. It uses advanced artificial intelligence technology to create an intelligent, efficient and accurate fast-moving consumer goods store inspection supervision method and technology, providing companies with better store inspection management solutions and enhancing their market competitiveness.
[0056] Example 2
[0057] The present invention also provides a store inspection supervision system based on multimodal data fusion, including a data acquisition module, a data processing module, a problem detection module and a report generation module.
[0058] During the store inspection, the data collection module collects multimodal data on shelf display, product placement, and customer flow. The multimodal data includes image data and voice data.
[0059] The data processing module processes multimodal data: it combines computer vision technology and natural language processing technology to extract the features of image data, and combines speech recognition technology and natural language processing technology to extract the audio features and semantic features of speech data.
[0060] The features of image data, audio features and semantic features of speech data are fused to obtain fused multimodal features. The multimodal neural network in the deep learning model is used to jointly train the recognition model based on the features of image data, audio features and semantic features of speech data to obtain the correlation information and mutual influence between the features.
[0061] The problem detection module uses the recognition model to detect problems based on the fused multimodal features.
[0062] The report generation module automatically generates a store inspection report based on the results of problem detection.
[0063] As the information interaction, execution process and other contents between the modules in the above-mentioned system are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.
[0064] Likewise, the benefits of the system of the present invention are:
[0065] Improve store inspection efficiency: Through automated data collection, processing and analysis, as well as intelligent problem detection and report generation, the store inspection process can be made more efficient, human resource consumption can be reduced, and time costs can be saved.
[0066] Improve store inspection accuracy: Combine multimodal data fusion technology, comprehensively consider multiple information sources such as images and voice, improve comprehensive understanding and accurate analysis of store conditions, reduce the impact of human factors, and improve the objectivity and accuracy of store inspection results.
[0067] Reduce store inspection costs: Reduce manual operation and management costs, improve store inspection efficiency and accuracy, thereby reducing the overall cost of store inspections and improving the competitiveness and profitability of the company.
[0068] Improve the intelligence level of store inspections: Use artificial intelligence technology to realize the automation and intelligence of the store inspection process, continuously optimize store inspection methods and techniques to enable them to adapt to changes in market demand, and improve store inspection management level and brand image.
[0069] It should be noted that not all steps and modules in the above-mentioned processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.
[0070] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.
Claims
1. A store inspection supervision method based on multimodal data fusion, characterized by include: Step 1: During the store inspection, multimodal data collection is performed on the shelf display, product placement, and customer flow in the store. The multimodal data includes image data and voice data. Step 2: Process multimodal data: Combine computer vision technology and natural language processing technology to extract the features of image data, and combine speech recognition technology and natural language processing technology to extract the audio features and semantic features of speech data. Step 3: Fuse the features of the image data, the audio features of the speech data, and the semantic features to obtain the fused multimodal features. Use the multimodal neural network in the deep learning model to jointly train the recognition model based on the features of the image data, the audio features of the speech data, and the semantic features to obtain the correlation information and mutual influence between the features. Step 4: Use the recognition model to detect problems based on the fused multimodal features. Step 5: Automatically generate a store inspection report based on the results of problem detection.
2. The store inspection supervision method based on multimodal data fusion according to claim 1 is characterized by: The multimodal data collection described in step 1 includes: deploying cameras, microphones, and sound sensors for the in-store environment, and collecting in-store image data and voice data through the cameras, microphones, and sound sensors respectively.
3. The method for store inspection supervision based on multimodal data fusion according to claim 1 is characterized in that the fusion of the features of the image data, the audio features and the semantic features of the speech data in step 3 comprises: Using the feature fusion formula: Perform feature fusion, where F fused is the fused multimodal feature vector, F i represents a certain modal eigenvector, W i It corresponds to the weight of a modal feature vector and is used to adjust the contribution of each modal feature.
4. The method for store inspection and supervision based on multimodal data fusion according to claim 1 is characterized in that Generating the store inspection report described in step 5 includes: generating the classification, description and improvement suggestions of the problem in the store inspection report according to the result of the problem detection, and adjusting and optimizing the recognition model according to the classification and description of the problem.
5. A store inspection supervision system based on multimodal data fusion, characterized by It includes data acquisition module, data processing module, problem detection module and report generation module. During the store inspection, the data collection module collects multimodal data on shelf display, product placement, and customer flow. The multimodal data includes image data and voice data. The data processing module processes multimodal data: it combines computer vision technology and natural language processing technology to extract the features of image data, and combines speech recognition technology and natural language processing technology to extract the audio features and semantic features of speech data. The features of image data, audio features and semantic features of speech data are fused to obtain fused multimodal features. The multimodal neural network in the deep learning model is used to jointly train the recognition model based on the features of image data, audio features and semantic features of speech data to obtain the correlation information and mutual influence between the features. The problem detection module uses the recognition model to detect problems based on the fused multimodal features. The report generation module automatically generates a store inspection report based on the results of problem detection.
6. The store inspection supervision system based on multimodal data fusion according to claim 5 is characterized by: The data acquisition module performs multimodal data acquisition, including: deploying cameras, microphones and sound sensors for the in-store environment, and collecting in-store image data and voice data through cameras, microphones and sound sensors respectively.
7. The store inspection supervision system based on multimodal data fusion according to claim 5 is characterized by: The data processing module integrates the features of image data, audio features and semantic features of speech data, including: Using the feature fusion formula: Perform feature fusion, where F fused is the fused multimodal feature vector, F i represents a certain modal eigenvector, W i It corresponds to the weight of a modal feature vector and is used to adjust the contribution of each modal feature.
8. The store inspection supervision system based on multimodal data fusion according to claim 5 is characterized by: The report generation module generates a store inspection report, including: generating the classification, description and improvement suggestions of the problem in the store inspection report according to the results of the problem detection, and adjusting and optimizing the recognition model according to the classification and description of the problem.