Repeated data removal method, system and equipment and medium

Abstract

The invention discloses a duplicated data removal method, system and device and a medium. The removal method comprises the steps that a first array, a second array and a count value are initialized; traversing the to-be-processed data, judging whether an identifier exists at a position corresponding to the current data in the first array or not, outputting the current data corresponding to an unidentified position as effective data, and identifying the corresponding position of the first array at the same time; wherein every time the valid data are output, the current valid data are synchronously stored in the second array in sequence, and one is added to the count value; and according to a plurality of valid data corresponding to the count value stored in the second array, completing identification emptying of a corresponding position in the first array, and returning the count value to zero at the same time. According to the invention, data deduplication can be rapidly realized through the first array, and the corresponding identification position can be rapidly found in the first array through the second array and the count value and emptying is completed, so that subsequent data can be continuously subjected to data deduplication.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and in particular relates to a method, system, device and medium for removing duplicate data. Background Art

[0002] At present, the 3D reconstruction technology based on line laser projects laser onto the surface of an object. The laser is modulated due to the change in the depth of the object surface and possible gaps. The light stripes change and become discontinuous in the image. The degree of change is proportional to the depth, and the discontinuity shows the physical gap on the surface of the object. This modulation process is reconstructed through a mathematical model, so that the 3D coordinates of the object are obtained based on the 2D laser stripe image of the intersection of the laser plane and the outer surface of the object. Due to its advantages of fast reconstruction speed, simple structure, high precision and strong anti-interference ability, it is widely used in many scenarios such as product size detection, weld tracking, workpiece contour measurement, robot trajectory guidance, etc. in various fields such as automobile production and electronic manufacturing.

[0003] Centerline cameras, also known as line laser 3D cameras, are 3D scanning devices based on line laser 3D reconstruction technology. The camera extracts the centerline of the line from the image and forms centerline data. In the process of outputting the centerline data in the current image, due to equipment failure, human error or other environmental factors, such as the centerline data being repeatedly copied or stored during transmission, there will be a large amount of duplicate data in the centerline data, so these duplicate data need to be removed. The centerline data mainly includes the point data of all centerlines in the current image (including coordinates x, y and intensity z), and the proportion of centerlines in a picture is usually small, and there are upper and lower limits for coordinates x and y. Therefore, the data range determined based on the image size is usually much larger than the effective data volume after the final deduplication.

[0004] Generally, the centerline data can be stored and deduplicated using a bitmap, but the insertion of the bitmap is slow and the clearing is also slow. If the sequential table is used for storage, the clearing is fast but difficult to find. Therefore, in order to combine the advantages of the above two methods, the deduplication speed and subsequent clearing can be presented in a relatively fast manner, and a duplicate data removal method, system, device and medium are provided. Summary of the invention

[0005] The purpose of the present invention is to overcome the above problems existing in the prior art and to provide a method, system, device and medium for removing duplicate data.

[0006] In order to achieve the above technical objectives and the above technical effects, the present invention is implemented through the following technical solutions: A method for removing duplicate data, wherein the duplicate data is a number of to-be-processed data with position information extracted through the line center, and the to-be-processed data is identified and removed based on the storage space, thereby obtaining valid data, and the storage space is cleared after the duplicate data is removed. The removal method includes: Initialize and set the first array, the second array and the count value, traverse the data to be processed, determine whether there is an identifier at the position corresponding to the current data in the first array, ignore the current data corresponding to the identified position, output the current data corresponding to the unidentified position as valid data, and simultaneously identify the corresponding position of the first array to identify whether the subsequent data to be processed is duplicate data; wherein, each time valid data is output, the current valid data is synchronously stored in sequence in the second array, and the count value is increased by one, so that the valid data in the second array corresponds to the identified position of the first array one by one; Based on the end mark of the data to be processed, and according to a number of valid data corresponding to the count value stored in the second array, the mark of the corresponding position in the first array is cleared, and the count value is reset to zero at the same time.

[0007] Furthermore, line center extraction includes: Collecting raw image data, and reading pixel values ​​in each column direction in parallel based on FPGA, wherein the raw image data includes line data distributed along the row direction; Determine the increasing and decreasing stages in the direction of each column to confirm the maximum value of each column; Based on the maximum value of each column, the position information and corresponding pixel values ​​of several adjacent pixels in the corresponding column direction are obtained, and the weighted centroid is calculated to obtain the line center point information in the direction of each column; The line center point information in each column direction is extracted in turn, and the output is summarized to form the data to be processed.

[0008] Furthermore, the size of the first array corresponds to the data range of the data to be processed.

[0009] Furthermore, a speed ratio of sequential reading to random reading is calculated to serve as a length ratio of the first array to the second array.

[0010] Further, determining whether there is an identifier at a position corresponding to the current data in the first array includes: Initialization setting: When initializing the first array, the second array and the count value, they are all reset to zero; Determine whether the position corresponding to the current data in the first array is 1: if it is 1, ignore the current data; if it is 0, set it to 1 and output the corresponding current data point as valid data.

[0011] Further, completing the clearing of the identifier of the corresponding position in the first array includes: when the end identifier is identified, determining whether the count value exceeds the size of the second array: If the count value does not exceed the size of the second array, a number of valid data corresponding to the count value in the front data of the second array are traversed, and the position corresponding to the mark 1 in the first array is confirmed based on the number of valid data, thereby completing the clearing of the first array; If the count value exceeds the size of the second array, directly traverse and clear all positions in the first array to 0.

[0012] Furthermore, after the count value returns to zero, new data to be processed is received again, and the above steps are repeated and cycled in sequence.

[0013] The present invention also provides a duplicate data removal system, comprising: An initialization module is used to initialize and set a first array, a second array, and a count value, wherein the first array is used to identify the corresponding position of the data to be processed to identify whether the subsequent data to be processed is duplicate data; the second array is used to store valid data to correspond to the position information identified in the first array; the count value is used to count the number of valid data to determine the storage status of valid data in the second array; wherein the valid data is non-duplicate data in the data to be processed, and the length of the second array is smaller than the first array; A data processing module is used to traverse the data to be processed, determine whether there is an identifier at a position corresponding to the current data in the first array, ignore the current data corresponding to the identified position, output the current data corresponding to the unidentified position as valid data, and identify the corresponding position of the first array at the same time; wherein, each time valid data is output, the current valid data is synchronously stored in sequence in the second array, and the count value is increased by one, so that the first array and the second array are synchronously updated; The data clearing module is used to clear the mark of the corresponding position in the first array based on the end mark of the data to be processed and to reset the count value to zero according to the valid data corresponding to the count value stored in the second array.

[0014] The present invention also provides an electronic device, comprising a storage medium and a processor, wherein the storage medium stores a computer program, and the processor implements the above removal method when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium, comprising a computer program, wherein the computer program implements the above-mentioned removal method when executed by a processor.

[0016] The beneficial effects of the present invention are: (1) The present invention can fully complete the deduplication processing of all the data to be processed by setting a larger first array, and can quickly identify whether the data is repeated based on the coordinate information in the data to be processed and the identification status of the corresponding storage position in the first array, and provide a data basis for subsequent deduplication identification through the subsequent continuous supplementation of identification. Since all the storage positions of the first array correspond to all the pixel positions in the image data corresponding to the data to be processed, any data to be processed can quickly find the corresponding storage position in the first array to achieve rapid indexing, so as to quickly confirm whether it is duplicate data, and the deduplication speed is fast. In order to achieve rapid clearing of the first array after deduplication is completed and to achieve continuous deduplication of subsequent data, the second array and the count value are introduced. According to the several valid data corresponding to the count value stored in the second array, the corresponding identification position can be quickly found in the first array and cleared. Since the data range determined based on the image size is usually much larger than the amount of valid data after the final deduplication, the second array can often be set to be much smaller than the length of the first array. Only a small amount of random read search is required in the first array to achieve rapid clearing, so as to provide continuous data deduplication for subsequent data.

[0017] (2) The present invention can be based on binary storage by judging whether the position corresponding to the current data in the first array is 1, so that the first array can quickly identify the identification state during the deduplication process, thereby improving the deduplication speed.

[0018] (3) The present invention can effectively improve the robustness of the present invention by judging whether the count value exceeds the size of the second array and the corresponding operation, and can effectively ensure that its efficacy can be kept stable under any special circumstances. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 It is a flow chart of the removal method in the present invention; Figure 2 It is a structural block diagram of the removal system in the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] like Figure 1 As shown, this embodiment first provides a method for removing duplicate data. Duplicate data is a number of to-be-processed data with position information after being extracted from the line center. The to-be-processed data is identified and duplicate data is removed based on the storage space, thereby obtaining valid data, and the storage space is cleared after the duplicate data is removed. The removal method specifically includes the following steps: Initialize and set the first array, the second array and the count value, traverse the data to be processed, determine whether there is an identifier at the position corresponding to the current data in the first array, ignore the current data corresponding to the identified position, output the current data corresponding to the unidentified position as valid data, and simultaneously identify the corresponding position of the first array to identify whether the subsequent data to be processed is duplicate data; wherein, each time valid data is output, the current valid data is synchronously stored in sequence in the second array, and the count value is increased by one, so that the valid data in the second array corresponds one to one with the identified position of the first array.

[0022] As a linear data structure, an array consists of elements of the same type, which are stored contiguously in memory. The characteristics of an array include fixed length and homogeneity. Fixed length means that once an array is created, its length is usually fixed; homogeneity means that all elements in an array must be of the same type, because the storage structure of the array requires that all elements occupy the same amount of memory space.

[0023] The first array is used as a storage space to identify the corresponding positions of the data to be processed, which means that all storage positions of the first array correspond one-to-one to all pixel positions in the image data corresponding to the data to be processed. At this time, any data to be processed can quickly find the corresponding storage position in the first array to achieve fast indexing, so as to quickly confirm whether it is duplicate data.

[0024] As a specific embodiment of the present invention, the data to be processed is streaming data. During the image acquisition process, the image data that needs to be continuously acquired will be generated in a continuous manner and usually needs to be processed immediately for timely response or analysis. Different data blocks will be distinguished by specific separators that will not appear in the data.

[0025] As can be seen from the above, duplicate data is a number of unprocessed data with position information after line center extraction. In order to efficiently and accurately find the line center information of the lines in the original image data, so as to cooperate with the subsequent rapid removal of duplicate data and improve efficiency as a whole, line center extraction includes: S1: Collecting raw image data, and reading pixel values ​​in each column direction in parallel based on FPGA, wherein the raw image data includes line data distributed along the row direction.

[0026] In the original image data, rows and columns are basic concepts used to describe the two-dimensional structure of the image. Rows usually correspond to the horizontal direction of the image, while columns correspond to the vertical direction of the image. In most image processing software and programming libraries, each row represents a horizontal scan line of the image, and each column represents a vertical scan line of the image. In the present invention, the distribution of lines along the row direction of the original image data means that at most one line section is passed through in the column direction, thereby ensuring the accuracy of line center extraction of the lines.

[0027] By creating a corresponding buffer area on the FPGA, the original image data can be stored row by row, and then multiple independent computing units are set to read the pixel values ​​in each column direction for subsequent calculations.

[0028] S2: Determine the increasing phase and decreasing phase in the direction of each column to confirm the maximum value of each column.

[0029] The pixel values ​​in each column direction are read in sequence, and a sliding window is introduced for processing. The size of the sliding window is L=2N+1. The extreme values ​​in each column direction are searched in sequence through the sliding window, that is, the pixel values ​​corresponding to the intersection of the increasing phase and the decreasing phase, and the largest extreme value is selected from several extreme values ​​as the column maximum value. If there are several identical pixel values ​​at the intersection of the increasing phase and the decreasing phase, they are symmetrically distributed along the sliding window to determine the boundary position of the sliding window.

[0030] S3: Based on the maximum value of each column, the position information and corresponding pixel values ​​of several adjacent pixels in the corresponding column direction are obtained, and the weighted centroid is calculated to obtain the line center point information in the direction of each column.

[0031] According to the maximum value of each column, the position O of the line center point in each column direction can be obtained by the following formula:

[0032] Among them, G q is the pixel value corresponding to each position of the sliding window, V q is the vertical coordinate data corresponding to each position of the sliding window, and q is the position information of the sliding window.

[0033] S4: Extract the line center point information in each column direction in turn, and summarize and output them to form the data to be processed.

[0034] At this point, all line center point information is pending data with position information, which is aggregated to form streaming data. Although there will be certain duplicate data in the subsequent transmission process, the existence of line center extraction greatly reduces the amount of data from the cardinality, so that the data range determined by the original image data is usually much larger than the effective data after final deduplication, that is, the actual storage ratio of the first array is much smaller than the total size of the first array. Therefore, combined with the subsequent steps two and three, it can be seen that it is of great significance to use the second array to index and determine the identification position of a small range in the large range of the first array.

[0035] In order to determine whether there is an identifier at a position corresponding to the current data in the first array, the following steps may be specifically included: Initialization setting: When initializing the first array, the second array and the count value, they are all reset to zero; Determine whether the position corresponding to the current data in the first array is 1: if it is 1, ignore the current data; if it is 0, set it to 1 and output the corresponding current data point as valid data.

[0036] As a specific implementation of the present invention, assuming that the data range interval of the data to be processed is [min, max], in order to enable the first array to fully identify all valid data, the size of the first array corresponds to the data range of the data to be processed, that is, the size of the first array can be max-min bytes, and an identifier corresponding to the pixel position corresponding to the valid data is recorded at each byte position in the first array.

[0037] Assuming that the data range interval of the to-be-processed data corresponds to the image data as W×H, the size of the first array may be W×H bytes, where W is the width of the image data and H is the height of the image data.

[0038] At this time, the corresponding pixel coordinate (i, j) can be determined according to the data information carried by the current data. The corresponding position of the current coordinate (i, j) in the first array is P=(W×i+j). For example, the image data corresponding to the data to be processed is 4096x2160, and the corresponding position of the current coordinate (1, 1) in the first array is P=(4096·1+1)=4097.

[0039] Each byte can be represented by 8 bits. The clearing operation of the first array can set each byte in the first array to (00000000). When it is necessary to determine whether the position corresponding to the current data in the first array is 1, if the highest bit of the corresponding byte is 1, the current data is ignored. If it is 0, the highest bit of the corresponding byte is set to 1, that is, (10000000). You only need to quickly analyze and adjust the highest bit data.

[0040] Based on the end mark of the data to be processed, and according to a number of valid data corresponding to the count value stored in the second array, the mark of the corresponding position in the first array is cleared, and the count value is reset to zero at the same time.

[0041] As a specific implementation of the present invention, completing the clearing of the identifier of the corresponding position in the first array includes: when the end identifier is identified, determining whether the count value exceeds the size of the second array: If the count value does not exceed the size of the second array, a number of valid data corresponding to the count value in the front data of the second array are traversed, and the position corresponding to the mark 1 in the first array is confirmed based on the number of valid data, thereby completing the clearing of the first array; If the count value exceeds the size of the second array, directly traverse and clear all positions in the first array to 0.

[0042] If the second array is full when valid data corresponding to the current count value is stored, it means that the count value exceeds the size of the second array; otherwise, it means that the count value does not exceed the size of the second array.

[0043] When the count value does not exceed the size of the second array, since the valid data is stored in the second array in order, all non-repetitive data in the current segment of data to be processed can be quickly extracted from the valid data corresponding to the count value in the front data of the second array according to the accumulated value of the current count value, and any non-repetitive data can be quickly located in the corresponding byte position in the first array through the above formula P=(W×i+j), so based on all the valid data in the second array, the corresponding byte position in the first array can be quickly confirmed. It can be seen from the above that the highest bit of the corresponding byte has been set to 1 at this time, and the position marked as 1 can be found according to the data analysis of the highest bit. Similarly, if the count value exceeds the size of the second array, it is impossible to accurately obtain a complete number of valid data to accurately locate the corresponding position in the first array, so it is sufficient to directly traverse and clear all positions in the first array to 0.

[0044] In order to continuously utilize the first array, the second array and the count value to remove duplicate data, after the count value returns to zero, new data to be processed may be received again, and the above steps may be repeated and cycled in sequence.

[0045] For ease of understanding, let the first array be M, the second array be A, and the count value be C. In order to reasonably set the length of the second array, the speed ratio of sequential reading and random reading can be calculated as the length ratio of the first array to the second array. The specific reasons are as follows: As can be seen from the above, when the count value does not exceed the size of the second array, the first array M is cleared based on the valid data in the second array A. Suppose the amount of valid data stored in the second array A is X, and X is less than or equal to the maximum amount of data that the second array A can store. At this time, the first array M is cleared at the random read speed R. 随 Reading X positions takes time T1=X / R 随 , and if the speed R is read in sequence 顺 Read the max-min positions in the first array M, and the time is T2=(max-min) / R 顺 , at least T1≤T2 must be satisfied at this time. It makes sense to clear the first array M through the second array A. Otherwise, directly traversing and clearing the first array M is the optimal solution. Therefore, substituting it into the formula yields: X / R 随 ≤ (max-min) / R 顺 That is X ≤ (max-min)·R 随 / R 顺 At this time, let the speed ratio of sequential reading and random reading be K, that is, R 顺 / R 随 = K, then X ≤ (max-min) / K, it can be seen that when X is at least less than or equal to (max-min) / K, it is valuable to clear the first array M through the second array A. However, the length of the second array A cannot be too small, otherwise the final count value C will exceed the size of the second array A. At this time, it is necessary to traverse and clear all positions of the first array M to 0, which will lose the function of the second array A. Therefore, in actual work, the length of the second array A can be adjusted based on experience, so that it can ensure that the count value C does not exceed the size of the second array A under the premise of a relatively small length. You can also set an empirical threshold B so that B ≤ X ≤(max-min) / K.

[0046] As can be seen from the above, the second array is used to store valid data, so the size of the second array is usually designed to be much smaller than the size of the first array M, that is, B ≤ the size of the second array A≤ (max-min) / K to make full use of limited resources. However, when resources are sufficient, the size of the second array A can be expanded so that the second array A can fully store all valid data, avoiding the situation where the count value C eventually exceeds the size of the second array A.

[0047] like Figure 2As shown, the second aspect of the present invention also provides a duplicate data removal system, comprising: An initialization module is used to initialize and set a first array, a second array, and a count value, wherein the first array is used to identify the corresponding position of the data to be processed to identify whether the subsequent data to be processed is duplicate data; the second array is used to store valid data to correspond to the position information identified in the first array; the count value is used to count the number of valid data to determine the storage status of valid data in the second array; wherein the valid data is non-duplicate data in the data to be processed, and the length of the second array is smaller than the first array; A data processing module is used to traverse the data to be processed, determine whether there is an identifier at a position corresponding to the current data in the first array, ignore the current data corresponding to the identified position, output the current data corresponding to the unidentified position as valid data, and identify the corresponding position of the first array at the same time; wherein, each time valid data is output, the current valid data is synchronously stored in sequence in the second array, and the count value is increased by one, so that the first array and the second array are synchronously updated; The data clearing module is used to clear the mark of the corresponding position in the first array based on the end mark of the data to be processed and to reset the count value to zero according to the valid data corresponding to the count value stored in the second array.

[0048] The specific operation steps of the initialization module, data processing module and data clearing module can refer to the above removal method.

[0049] The third aspect of the present invention further provides an electronic device, comprising a storage medium and a processor, wherein the storage medium stores a computer program, and the processor implements the above-mentioned removal method when executing the computer program.

[0050] A fourth aspect of the present invention further provides a computer-readable storage medium, comprising a computer program, wherein the computer program implements the above-mentioned removal method when executed by a processor.

[0051] In practical applications, the computer-readable storage medium may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device.

[0052] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0053] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0054] Computer program code for performing the operation of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0055] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0056] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A method for removing duplicate data, wherein the duplicate data is a number of to-be-processed data with position information after being extracted from the line center, and the to-be-processed data is identified and the duplicate data is removed based on the storage space, thereby obtaining valid data, and the storage space is cleared after the duplicate data is removed, characterized in that: Removal methods include: Initialize and set the first array, the second array and the count value, traverse the data to be processed, determine whether there is an identifier at the position corresponding to the current data in the first array, ignore the current data corresponding to the identified position, output the current data corresponding to the unidentified position as valid data, and simultaneously identify the corresponding position of the first array to identify whether the subsequent data to be processed is duplicate data; wherein, each time valid data is output, the current valid data is synchronously stored in sequence in the second array, and the count value is increased by one, so that the valid data in the second array corresponds to the identified position of the first array one by one; Based on the end mark of the data to be processed, according to a number of valid data corresponding to the count value stored in the second array, the mark of the corresponding position in the first array is cleared, and the count value is reset to zero at the same time.

2. A method for removing duplicate data according to claim 1, characterized in that: Line center extraction includes: Collecting raw image data, and reading pixel values ​​in each column direction in parallel based on FPGA, wherein the raw image data includes line data distributed along the row direction; Determine the increasing and decreasing stages in the direction of each column to confirm the maximum value of each column; Based on the maximum value of each column, the position information and corresponding pixel values ​​of several adjacent pixels in the corresponding column direction are obtained, and the weighted centroid is calculated to obtain the line center point information in the direction of each column; The line center point information in each column direction is extracted in turn, and the output is summarized to form the data to be processed.

3. A method for removing duplicate data according to claim 1, characterized in that: The size of the first array corresponds to the data range of the data to be processed.

4. The method for removing duplicate data according to claim 1, characterized in that: The speed ratio of sequential reading and random reading is calculated as the length ratio of the first array to the second array.

5. A method for removing duplicate data according to any one of claims 1 to 4, characterized in that: Determining whether there is an identifier at a position corresponding to the current data in the first array includes: Initialization setting: When initializing the first array, the second array and the count value, they are all reset to zero; Determine whether the position corresponding to the current data in the first array is 1: if it is 1, ignore the current data; if it is 0, set it to 1 and output the corresponding current data point as valid data.

6. A method for removing duplicate data according to claim 5, characterized in that: Completing the clearing of the identifiers at the corresponding positions in the first array includes: when the end identifier is identified, determining whether the count value exceeds the size of the second array: If the count value does not exceed the size of the second array, a number of valid data corresponding to the count value in the front data of the second array are traversed, and the position corresponding to the mark 1 in the first array is confirmed based on the number of valid data, thereby completing the clearing of the first array; If the count value exceeds the size of the second array, directly traverse and clear all positions in the first array to 0.

7. A method for removing duplicate data according to claim 6, characterized in that: After the count value returns to zero, new data to be processed is received again, and the above steps are repeated and cycled in sequence.

8. A duplicate data removal system, characterized in that: include: An initialization module is used to initialize and set a first array, a second array, and a count value, wherein the first array is used to identify the corresponding position of the data to be processed to identify whether the subsequent data to be processed is duplicate data; and the second array is used to store valid data to correspond to the position information identified in the first array; The count value is used to count the number of valid data to determine the storage of valid data in the second array; wherein the valid data is non-repetitive data in the data to be processed, and the length of the second array is smaller than the first array; A data processing module is used to traverse the data to be processed, determine whether there is an identifier at a position corresponding to the current data in the first array, ignore the current data corresponding to the identified position, output the current data corresponding to the unidentified position as valid data, and identify the corresponding position of the first array at the same time; wherein, each time valid data is output, the current valid data is synchronously stored in sequence in the second array, and the count value is increased by one, so that the first array and the second array are synchronously updated; The data clearing module is used to clear the mark of the corresponding position in the first array based on the end mark of the data to be processed and to reset the count value to zero according to the valid data corresponding to the count value stored in the second array.

9. An electronic device comprising a storage medium and a processor, wherein the storage medium stores a computer program, characterized in that: When the processor executes the computer program, the removal method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed by a processor, the removal method according to any one of claims 1 to 7 is implemented.

Images