A method for establishing an audiovisual database for garbage
Through multi-level data acquisition and matching methods, a garbage database containing visual, auditory and tactile data is established, which solves the problem of lack of data in garbage classification in the prior art, and improves the accuracy and stability of garbage identification.
Patent Information
- Application Number
- CN202111368460.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-18
- Publication Date
- 2025-02-18
- Estimated Expiration
- 2041-11-18
AI Technical Summary
The existing technology lacks objective conditions for large databases and batch collection of data in garbage classification, resulting in limited application of deep learning technology, especially the inability to accurately distinguish garbage with the same appearance but different materials.
A multi-level data acquisition method is adopted, including the collection and matching of visual, auditory and tactile data, to establish a spam database containing visual, auditory and tactile data, and to build an efficient database through data comparison, twinning and optimization methods.
The garbage audio-visual and touch database has been effectively established, which has solved the problem of serious lack of data in garbage identification research in the scientific research field, and improved the accuracy and stability of garbage identification.
Smart Images
Figure CN114117136B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an information collection and database establishment method, in particular to an audiovisual data collection method for garbage and a database establishment method, belonging to the field of garbage database establishment. Background Art
[0002] "How to reduce carbon emissions" has become a topic of great concern to people from all walks of life. At the current stage, effective classification and treatment of garbage sources is a relatively feasible means of energy conservation and emission reduction. However, there are two difficulties involved in this process.
[0003] The first difficulty is that the current garbage classification is mainly concentrated in the lives of residents in first-tier cities. There are currently few compulsory administrative measures for the classification of garbage. Based on this, scholars at home and abroad have begun to explore the problem of garbage source classification based on deep learning technology. Deep learning technology needs to be based on big data. At present, there is no large database on garbage available at home and abroad. What's more, relevant scholars often lack the objective conditions for batch collection of garbage data. This has caused a bottleneck in this research.
[0004] The second difficulty is that deep learning technology currently relies mainly on image data. However, image data often cannot accurately distinguish garbage with the same appearance but different materials, which makes classification technology that relies solely on image data have great limitations. Summary of the invention
[0005] In order to solve the above problems, the purpose of the present invention is to provide a multi-level data collection method and database establishment method for garbage, which collects and matches three types of data: visual (pictures), auditory (sounds), and tactile (sensor signals), effectively establishing an audio-visual and tactile database of garbage, providing a data basis for research and industrial applications.
[0006] In order to solve the above problems, the technical solution of the present invention includes:
[0007] Data collection method S1;
[0008] Effective information extraction method S2;
[0009] Data comparison method S3;
[0010] Data twin method S4;
[0011] Database construction method S5;
[0012] Data optimization method S6.
[0013] The data types in the database include visual data, auditory data, and tactile data of the same type of garbage. In particular, visual data refers to pictures; auditory data refers to sounds; and tactile data refers to tactile signals.
[0014] The data collection method S1 includes a batch big data collection method S1-1 and an experimental precise data collection method S1-2. Batch big data is obtained by the batch big data collection method S1-1, and experimental precise data is obtained by the experimental precise data collection method S1-2. The amount of data of the batch big data is greater than the amount of data of the experimental precise data, and the batch big data only has visual data, while the experimental precise data has complete visual data, auditory data, and tactile data.
[0015] The effective information extraction method S2 includes digitizing and extracting effective information of picture data in the data obtained by the data acquisition method S1, digitizing and extracting effective information of sound data, and digitizing and extracting effective information of tactile data.
[0016] The valid information of the image data includes but is not limited to category, name, material, shape, color, size, etc.
[0017] The effective information of the sound data includes but is not limited to pitch, volume, timbre, etc.
[0018] The effective information of the tactile data includes but is not limited to pressure, shape, temperature, hardness, roughness, thermal conductivity, sliding feeling, etc.
[0019] The data comparison method S3 is to compare only the valid information of the image data in the data processed by the valid information extraction method S2, and the comparison results are divided into three situations: different garbage, same garbage, and similar garbage.
[0020] Identical garbage refers to garbage with the same category, material, name, shape, and size; similar garbage refers to garbage with the same category, material, and name but different shapes and sizes; different garbage refers to situations other than identical garbage and similar garbage.
[0021] The data twin method S4 includes a precise twin method and an approximate twin method:
[0022] When the comparison result according to the data comparison method S3 is the same garbage, precise twin is used to process and obtain precise twin data; when the comparison result according to the data comparison method S3 is similar garbage, approximate twin is used to process and obtain approximate twin data.
[0023] The precise twinning method is:
[0024] The sound data and tactile data of a piece of garbage in the experimental precise data collected by the experimental precise data collection method S1-2 are copied to the image data with only the same valid information of the garbage image in the batch big data collected by the batch big data collection method S1-1 and combined to form a group of precise twin data groups, and then constitute precise twin data.
[0025] The approximate twin method is:
[0026] First, the image data of garbage in the batch big data collected by the batch big data collection method S1-1 and the image data of garbage in the batch big data collected by the experimental precise data collection method S1-2 are extracted respectively by the data comparison method S3 to form a database to be matched, so as to obtain a database to be matched of experimental precise data and a database to be matched of batch big data;
[0027] Then, the garbage is arranged in each database to be matched according to its size information, and the respective means and standard deviations are calculated. The ratio of the standard deviations of the two databases to be matched is used as the expansion ratio, and the database to be matched with a smaller data volume is copied and expanded according to the expansion ratio; after the expansion, the garbage is re-arranged in each database to be matched according to its size information;
[0028] Next, the two image data with the same arrangement position are matched by comparing them with the arrangement positions of the data in the two databases to be matched; after pairing, the sound data and tactile data of the garbage corresponding to the image data collected by the experimental precise data collection method S1-2 are copied to the image data of the garbage corresponding to the image data collected by the batch big data collection method S1-1 and combined to form a group of approximate twin data groups, and then constitute approximate twin data.
[0029] The database construction method S5 includes two parts of data. The first part of the data is the precise twin data and approximate twin data obtained by the data twin method S4; the second part of the data is the experimental precise data collected by the experimental precise data collection method S1-2.
[0030] The data optimization method S6 includes gradually optimizing only the approximate twin data, and the optimization method is:
[0031] The approximate twin data and the experimental precise data newly collected by the experimental precise data collection method S1-2 are compared cyclically in turn for effective image information. Only the effective image information is compared. If the comparison result is the same garbage, the approximate twin data is reorganized by the precise twin method. That is, the auditory data and tactile data in the experimental precise data belonging to the same garbage are replaced by the auditory data and tactile data in the original approximate twin data.
[0032] The beneficial effects of the present invention are:
[0033] The present invention can effectively establish an effective and accurate audio-visual database for garbage, and effectively solve the serious data shortage problem faced by garbage identification research in the scientific research field. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 The following is a flow chart of the overall method;
[0035] Figure 2 Build a method flow logic diagram for the audio-visual touch database;
[0036] Figure 3 Provide a logical diagram of the data optimization method flow;
[0037] Figure 4 This is a sample picture of “same garbage”;
[0038] Figure 5 This is a sample picture of “similar garbage”;
[0039] Figure 6 This is an example of “similar garbage”. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the following is a detailed description with reference to the accompanying drawings and embodiments. It should be understood that the invention examples specifically described herein are only used to explain the present invention and are not limited to the present invention.
[0041] Embodiments of the present invention Figures 1 to 3 As shown, the details are as follows:
[0042] Data collection method S1:
[0043] like Figure 1 As shown, the data collection method S1 includes a batch big data collection method S1-1 and an experimental precise data collection method S1-2. Batch big data is obtained by the batch big data collection method S1-1, and experimental precise data is obtained by the experimental precise data collection method S1-2. The data volume of the batch big data is inconsistent with the data volume of the experimental precise data. The data volume of the batch big data is larger than that of the experimental precise data, and the batch big data only has visual data, while the experimental precise data has complete visual data, auditory data, and tactile data.
[0044] The specific implementation of batch big data collection method S1-1 is to extract from a known image database or obtain from a web crawler.
[0045] Experimental precise data acquisition method S1-2 is collected using an experimental precise data acquisition system.
[0046] Effective information extraction method S2:
[0047] like Figure 2 As shown, the effective information extraction method S2 includes digitizing and extracting effective information of picture data in the data obtained by the data acquisition method S1, digitizing and extracting effective information of sound data, and digitizing and extracting effective information of tactile data.
[0048] The first part is the pictures in the collected batch big data, which are named Picture(i), where i is the number. The effective information of Picture(i) is extracted, including but not limited to category, shape, color, material, name, size, etc.
[0049] The second part is the precise experimental data, including pictures, sounds, tactile data, etc., which are named Photo(s), Voice(s), Touch(s), where s is a number. The effective information extracted from the picture Photo(s) includes but is not limited to category, shape, color, material, name, size, etc.; the effective information extracted from the voice Voice(s) includes but is not limited to pitch, volume, timbre, etc.; the effective information extracted from the tactile data Touch(s) includes but is not limited to pressure, temperature, roughness, sliding sensation, thermal conductivity, etc.
[0050] Data comparison method S3:
[0051] like Figure 2 As shown, the data comparison method S3 is to compare only the effective information of the image data in the data processed by the effective information extraction method S2, and the comparison results are divided into three cases: different garbage, the same garbage, and similar garbage. In particular, the comparison method is: circular comparison. In particular, the comparison order is: category-material-name-shape-size.
[0052] Data comparison method S3( Figure 2 ) refers to the cyclic comparison of effective information of pictures, in particular, only the effective information of pictures is compared. The pictures involved in the comparison are the pictures of batch big data Picture(i) and the pictures of experimental precise data Photo(s). The comparison is in the form of cyclic comparison, and the contents of the comparison involve categories, materials, names, shapes, sizes, etc. According to the comparison results, it is judged whether the two belong to the same garbage, similar garbage or different garbage.
[0053] If any of the information in terms of category, material, or name is different, the next Photo(s) will be replaced for comparison again; if the category, material, and name are the same, the shapes and sizes of the two will be further compared to see if they are the same. If the shapes and sizes are still the same, the garbage involved in the two photos Picture(i) and Photo(s) involved in the comparison is deemed to be the same garbage; if any of the shapes and sizes are different, the garbage involved in the two photos Picture(i) and Photo(s) involved in the comparison is deemed to be similar garbage.
[0054] Implementation example Figure 4-Figure 6 shown.
[0055] Figure 4 These are sample images of the same garbage, with the left image showing a Coca-Cola aluminum can and the right image showing a Coca-Cola aluminum can.
[0056] (1) Figure 4 The middle left picture comes from precise experimental data, and the sound data and tactile data corresponding to the picture are all from the precise experimental collection system; the right picture comes from batch big data (collected by mobile phone), which only has image data, and no auditory and tactile data.
[0057] (2) Yes Figure 4 The two pictures were compared in terms of "category, material, name, shape, and size" and the comparison results were exactly the same, so the two pieces of garbage were identified as "the same garbage."
[0058] (3) Figure 4 The auditory and tactile information corresponding to the middle left picture is copied to the right picture for precise twinning.
[0059] Likewise, Figure 5 These are sample images of similar garbage, with the left image showing a Coca-Cola aluminum can and the right image showing a Coca-Cola aluminum can (small size).
[0060] (1) Figure 5 The middle left picture comes from precise experimental data, and the sound data and tactile data corresponding to the picture are all from the precise experimental collection system; the right picture comes from batch big data (collected by mobile phone), which only has image data, and no auditory and tactile data.
[0061] (2) Yes Figure 5 The two pictures were compared in terms of "category, material, name, shape, and size" and found to be consistent in "category, material, and name" but different in "shape" and "size". Therefore, the two pieces of garbage were identified as "similar garbage".
[0062] Likewise, Figure 6These are sample images of similar garbage, with the left image showing a Coca-Cola aluminum can and the right image showing a Fanta aluminum can.
[0063] (1) Figure 6 The middle left picture comes from precise experimental data, and the sound data and tactile data corresponding to the picture are all from the precise experimental collection system; the right picture comes from batch big data (collected by mobile phone), which only has image data, and no auditory and tactile data.
[0064] Data Twin Method S4:
[0065] like Figure 2 As shown, data twin method S4 includes precise twin method and approximate twin method:
[0066] When the comparison result according to the data comparison method S3 is the same garbage, precise twin is used to obtain precise twin data; when the comparison result according to the data comparison method S3 is similar garbage, approximate twin is used to obtain approximate twin data.
[0067] The precise twin method is: copy the sound data and tactile data of a piece of garbage in the experimental precise data collected by the experimental precise data collection method S1-2 to the image data with only the same valid information of the garbage image in the batch big data collected by the batch big data collection method S1-1 and combine them. Only the valid information of the garbage image is the same means that the comparison result is the same garbage, forming a group of precise twin data groups, and then all garbage constitutes precise twin data.
[0068] The approximate twin method is as follows: first, the image data of garbage in the batch big data collected by the batch big data collection method S1-1 and the image data of garbage in the batch big data collected by the experimental precise data collection method S1-2 are extracted separately through the data comparison method S3 to obtain all the garbage with the comparison results of similar garbage to form a database to be matched, so as to obtain a database to be matched of the experimental precise data and a database to be matched of the batch big data;
[0069] Then, the garbage is arranged in each database to be matched according to its size information, and the respective means and standard deviations are calculated. The ratio of the standard deviations of the two databases to be matched is used as the expansion ratio, and the database to be matched with a smaller data volume is copied and expanded according to the expansion ratio; after the expansion, the garbage is re-arranged in each database to be matched according to its size information;
[0070] Next, the two image data with the same arrangement position are matched by comparing them with the arrangement positions of the data in the two databases to be matched; after matching, the sound data and tactile data of the garbage corresponding to the image data collected by the experimental precise data collection method S1-2 are copied to the image data of the garbage corresponding to the image data collected by the batch big data collection method S1-1 and combined to form a group of approximate twin data groups, and then all garbage constitutes approximate twin data.
[0071] The comparison results show different garbage situations and no processing is done.
[0072] In the specific implementation, if the two garbages are judged to be similar garbages through the pictures, the experimental precise data will be copied to the "Photo(s) database to be matched", and at the same time, the picture data of the batch big data will be copied to the "Picture(i) database to be matched". Then the picture data of the two databases to be matched are sorted according to the size information of the garbage, and the "Photo(s) database to be matched" with a smaller data volume is expanded until the data volume of the two databases to be matched is the same. After expansion, the picture data in the two databases to be matched are still sorted according to the size information of the garbage. After sorting, the picture data at corresponding positions in the two databases to be matched are paired respectively. After pairing, the auditory data and tactile data corresponding to the picture data in the "Photo(s) database to be matched" are copied to the picture data at corresponding positions in the "Picture(i) database to be matched" in order to form multiple sets of new audio-visual and tactile data. This twin method is defined as approximate twin.
[0073] Specifically, when it is determined that the garbage involved in photos Picture(i) and Photo(s) is the same garbage, the Voice(s) and Touch(s) corresponding to Photo(s) are copied, and the copied Voice(s), Touch(s) and Picture(i) form a new audio-visual touch data set (precise twin data). The implementation method of the approximate twin method is as follows: when it is determined that the garbage involved in photos Picture(i) and Photo(s) is similar garbage, the image data collected from the experimental terminal is copied to the "Photo(s) database to be matched", and at the same time, the image data collected from the student group is copied to the "Picture(i) database to be matched". Then, a size median is taken from each of the two databases to be matched as the median of the normal distribution, and they are arranged according to the normal distribution, and their respective means μ and standard deviations σ are determined. The ratio of the standard deviation σ(i) and σ(s) of the two databases to be matched is used as the expansion ratio of the database to be matched Photo(s). The database to be matched Photo(s) is expanded according to the expansion ratio until the data volume of the two databases to be matched is the same. After expansion, the image data in the two databases to be matched are still sorted according to the size information of the garbage. After sorting, according to the normal distribution position of each data in the two databases, the corresponding positions of Picture(i) and Photo(s) are paired. After pairing, the voice information Voice(s) corresponding to the picture Photo(s) is
[0074] The Voice(s) and Touch(s) information are copied and the copied Voice(s) and Touch(s) information are distributed according to the normal distribution formula
[0075] Perform numerical optimization and match it to Picture(i) after optimization to form multiple sets of new audio-visual and touch data sets (approximate twin data).
[0076] Database construction method S5:
[0077] Implementation method of database construction method S5 ( Figure 2 ) is: add and fuse the "audio-visual-touch twin database" and the experimental precise data collected by the experimental precise data collection method S1-2 to construct the "audio-visual-touch total database".
[0078] Therefore, the junk data created by the present invention according to the above method can simultaneously meet the requirements of large data volume and high efficiency and accuracy, and can be used for data training and processing.
[0079] The research method of garbage classification based solely on image data is not ideal in accuracy, and the recognition process is easily restricted by the external environment (such as light). The present invention uses multiple types of data such as vision, hearing and touch for fusion analysis, which can effectively improve the accuracy of garbage recognition and at the same time improve the stability of recognition.
[0080] The garbage classification method requires rich data support. Among them, the data that is easier to obtain is image data, while sound data and tactile data require professional equipment to extract. The method provided by the present invention can effectively solve the problems of data type, quantity and accuracy. It can provide stable and effective database support, realize the multimodal stability and accuracy of garbage classification, improve the accuracy of garbage classification, and provide a data basis for related research.
[0081] like Figure 2 As shown, the database construction method S5, the data in the total database has two sources, including two parts of data. The first part of the data is the precise twin data and approximate twin data obtained by the data twin method S4, and the precise twin data and the approximate twin data together constitute the "audio-visual and touch twin database"; the second part of the data is the experimental precise data collected by the experimental precise data collection method S1-2, that is, the audio-visual and touch experimental data.
[0082] Data optimization method S6:
[0083] like Figure 3 As shown, the data optimization method S6, including the step-by-step optimization of only the approximate twin data, can significantly improve the accuracy of the database. The optimization method is:
[0084] Approximate twin data and experimental precise data collection method S1-2 The newly collected experimental precise data (newly added data) are cyclically compared with the effective information of the pictures in turn, and only the effective information of the pictures is compared, and the comparison content includes category, material, name, shape, and size; if the comparison result is the same garbage, the precise twin method is used to reorganize the approximate twin data, that is, the auditory data and tactile data in the experimental precise data belonging to the same garbage are replaced with the auditory data and tactile data in the original approximate twin data. If the comparison result shows that the garbage involved in the two is similar garbage, the group of data will not be updated.
[0085] In this way, if the valid information involved in the comparison is the same, it is identified as the same garbage, and a second precise twin is performed to replace the original approximate twin data, thereby optimizing the accuracy of the database.
Claims
1. A method for establishing an audiovisual database for junk, characterized in that The methods include the following: Data collection method S1; Effective information extraction method S2; Data comparison method S3; Data twin method S4; Database construction method S5; Data optimization method S6; The data collection method S1 includes a batch big data collection method S1-1 and an experimental precise data collection method S1-2. The batch big data is obtained by the batch big data collection method S1-1, and the experimental precise data is obtained by the experimental precise data collection method S1-2. The amount of data in the batch big data is greater than the amount of data in the experimental precise data, and the batch big data only has visual data, while the experimental precise data has complete visual data, auditory data, and tactile data. The data twin method S4 includes a precise twin method and an approximate twin method: When the comparison result according to the data comparison method S3 is the same garbage, the precise twin is used to process and obtain the precise twin data; when the comparison result according to the data comparison method S3 is similar garbage, the approximate twin is used to process and obtain the approximate twin data; The precise twinning method is: The sound data and tactile data of a piece of garbage in the experimental precise data collected by the experimental precise data collection method S1-2 are copied to the picture data with only the same valid information of the garbage picture from the batch big data collected by the batch big data collection method S1-1 and combined to form a set of precise twin data groups, thereby forming precise twin data; The approximate twin method is: First, the image data of garbage in the batch big data collected by the batch big data collection method S1-1 and the image data of garbage in the batch big data collected by the experimental precise data collection method S1-2 are extracted respectively by the data comparison method S3 to form a database to be matched, so as to obtain a database to be matched of experimental precise data and a database to be matched of batch big data; Then, the garbage is arranged in each database to be matched according to its size information, and the respective means and standard deviations are calculated. The ratio of the standard deviations of the two databases to be matched is used as the expansion ratio, and the database to be matched with a smaller data volume is copied and expanded according to the expansion ratio; after the expansion, the garbage is re-arranged in each database to be matched according to its size information; Next, the arrangement positions of the data in the two databases to be matched are compared, and the two image data in the same arrangement position are paired; After pairing, the sound data and tactile data of the garbage corresponding to the image data collected by the experimental precise data collection method S1-2 are copied to the image data of the garbage corresponding to the image data collected by the batch big data collection method S1-1 and combined to form a group of approximate twin data groups, and then constitute approximate twin data.
2. The method for establishing a database of audiovisual information for garbage according to claim 1, characterized in that: The data types in the database include visual data, auditory data, and tactile data of the same type of garbage.
3. The method for establishing a database of audiovisual information for garbage according to claim 1, characterized in that: The effective information extraction method S2 includes digitizing and extracting effective information of picture data in the data obtained by the data acquisition method S1, digitizing and extracting effective information of sound data, and digitizing and extracting effective information of tactile data.
4. The method for establishing a database of audiovisual information for garbage according to claim 1, characterized in that: The data comparison method S3 is to compare only the valid information of the image data in the data processed by the valid information extraction method S2, and the comparison results are divided into three situations: different garbage, same garbage, and similar garbage.
5. The method for establishing a database of audiovisual information for garbage according to claim 1, characterized in that: The database construction method S5 includes two parts of data. The first part of the data is the precise twin data and approximate twin data obtained by the data twin method S4; the second part of the data is the experimental precise data collected by the experimental precise data collection method S1-2.
6. The method for establishing a database of audiovisual information for garbage according to claim 1, characterized in that: The data optimization method S6 includes gradually optimizing only the approximate twin data, and the optimization method is: The approximate twin data and the experimental precise data newly collected by the experimental precise data collection method S1-2 are compared cyclically in turn for effective image information. Only the effective image information is compared. If the comparison result is the same garbage, the approximate twin data is reorganized by the precise twin method. That is, the auditory data and tactile data in the experimental precise data belonging to the same garbage are replaced by the auditory data and tactile data in the original approximate twin data.
Citation Information
Patent Citations
System for managing population health risks of municipal domestic waste incineration disposal facilities
CN107563646A
Garbage classification software based on computer vision
CN110929760A