Unlock instant, AI-driven research and patent intelligence for your innovation.

Message duplicate removal method and related device

A message and message reading technology, applied in the computer field, can solve the problem of low accuracy of message deduplication

Pending Publication Date: 2021-09-07
上海微盟企业发展有限公司
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of this application is to provide a message deduplication method, a message deduplication device, a server and a computer-readable storage medium to solve the problem of low message deduplication accuracy

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Message duplicate removal method and related device
  • Message duplicate removal method and related device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] The core of the present application is to provide a message de-duplication method, a message de-duplication device, a server and a computer-readable storage medium, so as to solve the problem of low message de-duplication accuracy.

[0040] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

[0041] In related technologies, Bloom filters are generally used for deduplication processing, or external key-v...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a message duplicate removal method. The method comprises the steps that a message production node of a distributed system writes a message to a first set; when the message consumption node of the distributed system reads the message of the second set, the message production node writes the message to the second set, so that the message consumption node reads the message from the first set, wherein the first set and the second set are sets of data structures without duplicate values. Through the switching operation between the first set and the second set and the data structure characteristics of the sets, deduplication is carried out on the messages in the sets, and the accuracy of message deduplication is improved. The invention further discloses a message duplicate removal device, a server and a computer readable storage medium, which have the above effects.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular to a message deduplication method, a message deduplication device, a server, and a computer-readable storage medium. Background technique [0002] With the continuous development of information technology, more and more messages are produced on the Internet, and more and more nodes or devices consume messages. Efficient management of transmitted messages is required. Furthermore, in the face of a large number of messages, repeated messages are likely to appear, which not only increases the burden on the hardware, but also leads to redundant sending of messages. Therefore, it is necessary to deduplicate the messages. [0003] In related technologies, Bloom filters are generally used for deduplication processing, or external key-value (keyword-value) storage is introduced for deduplication processing. However, due to the ever-increasing amount of data, there are som...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/215G06F16/27
CPCG06F16/215G06F16/27
Inventor 李浩
Owner 上海微盟企业发展有限公司