31results about How to "Improve the efficiency of duplicate checking" patented technology

The invention discloses a method and system for eliminating duplication during data count as well as a server and a storage medium, applicable to data duplication elimination in big data. The method provided by the invention comprises the following steps: receiving a call request, and performing load balancing by utilizing a dubbo component; analyzing the request, and according to a preset duplication elimination rank parameter in the request, creating a corresponding quantity of redis data storage bitmaps on the server; and acquiring a duplication elimination content parameter and the duplication elimination rank parameter in the request, calculating by virtue of a Bloom Filter algorithm to obtain a duplication elimination result, when the duplication elimination rank is higher than grade1 and a duplication elimination result return value is 0, calculating one group of hash functions again, and then performing duplication elimination again by virtue of the Bloom Filter algorithm. Inthe method disclosed by the invention, the load balancing is performed by virtue of the dubbo component, and according to the preset duplication elimination grade, count duplication elimination at a corresponding grade is performed by virtue of the Bloom Filter algorithm, so that data can be efficiently and rapidly processed, the probability that the data is eliminated mistakenly can be greatly reduced, and duplication elimination accuracy is improved.

The embodiment of the invention discloses a video duplicate checking method and device, and the method comprises the steps: constructing a multi-modal feature vector of a to-be-processed video; carrying out neighbor retrieval in a video library based on the multi-modal feature vector, screening out candidate videos similar to the to-be-processed video, and obtianing a candidate video set; calculating the similarity between each candidate video and the to-be-processed video to obtain a similarity result; and determining whether the to-be-processed video passes duplicate checking detection or not according to the similarity result. According to the scheme, the video duplicate checking efficiency can be improved while the video duplicate checking accuracy is ensured.

The invention relates to a medical data duplicate checking and associating method and system. The method comprises the following steps of: (1) extracting core data items in to-be-processed medical data; (2) classifying the core data items; (3) respectively carrying out preliminary screening on the data items in an exclusion array and a fuzzy array; (4) carrying out deep screening on the data items in the core data items; (5) setting a threshold value M2 of a suspected duplicated data similarity and / or a threshold value M3 of suspected associated data; and (6) after artificially checking and judging the suspected duplicated and / or associated data, inputting the data which is judged as non-duplicated data into a medical database, and endowing the data which is judged to be associated with one or more corresponding association labels. Compared with the prior art, the method and system provided by the invention have the characteristics of being low in missed judging rate, low in wrong judging rate and high in duplicate checking efficiency, and do not have high requirement for the profession degree of artificial checking, so that the operation costs of the duplicate checking and associating are remarkably reduced.

The invention relates to artificial intelligence, and discloses a project duplicate checking method, device and equipment and a storage medium, and the method comprises the steps: obtaining a projecttext, and dividing the project text into a to-be-detected short text set and a to-be-detected long text set; searching a reference short text corresponding to the to-be-detected short text set, and obtaining a first similarity between the reference short text and the to-be-detected short text set; if the first similarity is lower than a preset similarity threshold, searching a reference long textcorresponding to the to-be-measured long text set and obtaining a second similarity between the reference long text and the to-be-measured long text set; obtaining a duplicate checking result according to the second similarity, according to the invention, performing similarity detection on the short text set according to the reference short text corresponding to the short text set; when the obtained similarity cannot judge the duplicate checking condition of the project, judging the duplicate checking result of the project to be subjected to duplicate checking by calculating the similarity between the long text set and the reference long text, and compared with an existing text duplicate checking mode, the duplicate checking result is more accurate and real, and the text duplicate checkingefficiency is also improved.

The embodiment of the invention relates to a text data collection method and device. The collection method comprises the steps of performing duplicate checking on a text database based on a first Hashvalue obtained by calculating a text fragment with a set character length in a first target text through a first Hash algorithm; if the duplicate checking is not hit, storing the first target text into the text database, and configuring the text type of the first target text in the text database as a first type; Selecting a second target text from the text of which the text type is the first typein the text database, and performing duplicate checking on the text database based on a second hash value calculated from the second target text through a second hash algorithm; and if the duplicatechecking is missed, changing the text type of the second target text in the text database into a second type, or else deleting the data corresponding to the second target text from the database in thetext database. According to the text data duplicate checking method and device, text data can be efficiently subjected to duplicate checking based on different Hash algorithms in the text data collection process.

A method for checking duplication of text is disclosed, A fingerprint sequence of duplicate text to be checked can be stored in a text fingerprint database in advance, After the target text is obtained, the target fingerprint sequence is generated, and then the similar fingerprint sequence of each fingerprint in the target fingerprint sequence is calculated to obtain the similar fingerprint sequence. Finally, the fingerprint sequence including the target fingerprint sequence or the similar fingerprint sequence in the text fingerprint database is determined, and obviously, the text corresponding to the fingerprint sequence is the text similar to the target text. It can be seen that the method can generate similar fingerprint sequences of target fingerprint sequences, When judging whether the duplicated text and the target text are similar, only the fingerprint sequence of the duplicated text to be checked can be judged whether the fingerprint sequence of the duplicated text includes thetarget fingerprint sequence or the similar fingerprint sequence, and the similarity calculation of the duplicated text and the target text is not needed, thus saving the calculation amount and improving the duplication checking efficiency of the text. In addition, the present application also provides a text duplication checking apparatus, an apparatus, and a computer-readable storage medium, thefunctions of which correspond to the functions of the above-described method.

The embodiment of the invention discloses a thesis duplicate checking method and device, equipment and storage media. The thesis duplicate checking method comprises the following steps that: in a duplicate checking display interface, providing at least two optional duplicate checking platforms and the introduction information of each optional duplicate checking platform for a user to select as least one duplicate checking platform from the at least two optional duplicate checking platforms as a standby duplicate checking platform according to requirements, wherein the introduction information contains a charging standard, and the standby duplicate checking platform shares the thesis provided by the user; and according to the thesis and the charging standard of each standby duplicate checking platform, determining a payment amount, and finishing payment in one time to enable each standby duplicate checking platform to start a duplicate checking operation. By use of the embodiment of the invention, at least two optional duplicate checking platforms are provided for users to carry out selection, the standby duplicate checking platform shares the thesis provided by the user, payment is finished in one time during payment, so that the user does not need to submit the thesis on a plurality of duplicate checking platforms, multiple payment is carried out so as to save thesis duplicate checking time, and duplicate checking efficiency is improved.

The invention relates to a client information duplication checking method. The client information duplication checking method comprises the follow steps: firstly, client information is stored in a client information table in a database; then one or more keywords corresponding to each client information item are respectively generated and are stored in a keyword table; finally, a duplication checking operation is carried out in the database according to the keywords. The invention further relates to a duplication checking system corresponding to the duplication checking method. The client information duplication checking method and the client information duplication checking system largely improve the client information duplication checking efficiency, the duplication checking result is comprehensive, and repeated clients can not be missed.

The invention discloses a book joint selection method and system, and the method comprises the steps: a book selection management system is set in a library, a new book list database and a formulatedorder library are established in the book selection management system, and all publishing houses and readers upload recommended book list information to the new book list database; the book selectionmanagement system performs data integration and duplicate checking on the booklist information, and imports the booklist information subjected to duplicate checking into a planned order library; the library administrator draws the booklist information of the drafted order library and generates a drafted order, and the drafted order library sends the drafted order to a local database of the library; and the local database performs data duplicate checking on the proposed order again and generates a book issuing order, and finally sends the book issuing order to a publishing house to order books.The book selection management system is in butt joint with the local system of the library and the inventory system of the publishing house, a book selection, sharing and service platform is built, and book retrieval, duplicate checking, ordering and distribution integrated service is provided.

The invention discloses an electronic invoice duplicate checking method and system, and belongs to the technical field of electronic invoices, and the method comprises the steps: obtaining the title information of a to-be-detected electronic invoice; Analyzing the head-up information of the electronic invoice to be detected to obtain at least one of an invoice code, an invoice number, an amount, an invoice issuing date and a check code; Taking the analyzed information as a unique identifier, firstly comparing the electronic invoice title information with all invoices in an invoice input page,and then comparing the analyzed unique identifier with a unique identifier in a data table pre-stored in a database; And when any comparison result is the same, determining that the repeated electronic invoices exist. According to the method, whether the electronic invoice is reimbursed repeatedly or not is automatically verified before the electronic invoice is input, so that the duplicate checking efficiency of the electronic invoice is greatly improved.

The invention provides a duplicate checking method. The method comprises the following steps of the first step, using a Word2Vec model to train to obtain a sentence vector of an original sentence anda contrast sentence, wherein the sentence vector is obtained by integrating a word vector and a word characteristic vector; the second step, based on the sentence vector of the original sentence and the sentence vector of the contrast sentence, calculating to obtain the included angle between the sentence vector of the original sentence and the sentence vector of the contrast sentence; the third step, determining the similarity between the original sentence and the contrast sentence, wherein when the included angle is less than or equal to a threshold, it is determined that the original sentence is similar to the contrast sentence; when the angle is greater than the threshold, it is determined that the original sentence is not similar to the contrast sentence. The method comprehensively considers the word vector and the word characteristic vector, compared with the calculation time based on sentence coding, the calculation time of the method is obviously shortened, the introduction ofthe word characteristic vector has a certain solution effect on the situation that checking is difficult after synonym replacing, the problem that word changing, order changing and adding or deletionof words are difficult to check based on complete sentence comparison is solved, and on the whole, the method not only improves the accuracy of duplicate checking, but also improves the efficiency ofthe duplicate checking.

The invention discloses a device information writing control method and system, and the method comprises the following steps: a device information generation step: based on the number N of to-be-written devices and a customized character string, registering to generate N device SN and MAC addresses, N being a positive integer greater than or equal to 1; a database uploading step: uploading the N SNs into a first database table of the database, and recording the state of the SNs as an inactive state in the first database table; an online searching step: scanning a bar code on a device to be written, reading an SN and an MAC address to be confirmed, and respectively comparing the SN and the MAC address to be confirmed with the SN in the first database table and the historical MAC address in the second database table one by one to obtain a confirmed SN and an MAC address; and an information writing step: connecting the to-be-written device, and writing the confirmed SN and the MAC address into the to-be-written device to obtain an information writing completion device. The method provided by the invention can better control the legality of the SN and MAC address write-in equipment.

An embodiment of the invention provides an object uploading method. The method of a network side comprises the steps of receiving a duplicate checking request sent by a user terminal, wherein the duplicate checking request carries a data size of a specified object and a specified container identifier; searching for a first class object with the same data size as the specified object in a storage container corresponding to the specified container identifier; and if the first class object is not found, returning a first duplicate checking response to the user terminal, wherein the first duplicate checking response is used for instructing the user terminal to upload the specified object. According to the method, the duplicate checking efficiency and the object uploading efficiency of the user terminal can be improved. The invention furthermore provides an electronic device of the network side, an object uploading method of a user terminal side and the user terminal.

The invention provides a picture duplicate checking method and device and a computer readable storage medium, and the method comprises the steps: generating a corresponding character string for a picture to be subjected to duplicate checking based on the attribute information of the picture to be subjected to duplicate checking; performing picture duplicate checking calculation on the character string through a Bloom filter; and outputting a duplicate checking result through the Bloom filter. According to the technical scheme, the picture duplicate checking process is simplified, and the duplicate checking efficiency is improved.

The invention discloses a project duplicate checking method and system based on concurrent tasks. The method comprises four steps of carrying out the dynamic analysis on the Internet hot words and common words through the Internet technology, and forming a cloud lexicon; and matching the text information in the declaration material with a cloud word bank through a text matching method, segmentingthe declaration material into the semantic word segmentation factors, obtaining an optimal word segmentation scheme through weighted calculation, counting the word frequency and eliminating the high-frequency single words; and returning the similarity values of the current duplicate checking projects and the historical projects to the segmented word subset of the current duplicate checking projectand the segmented word subset of the historical project through a cosine similarity algorithm CosineSimilar. During the big data calculation, a high-capacity high-speed memory is utilized, the memorymanagement is reasonably used, the frequent read-write access of a hard disk is reduced, the concurrent multithreading tasks are started, the system resources are fully utilized, and the maximum frequency of a CPU is brought into play, so that the duplicate checking efficiency is improved.

The embodiment of the invention provides an article duplicate checking method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: inputting a feature vector of an article to be detected into a pre-trained duplicate checking model to obtain a related article related to the article to be detected, and determining a duplicate checking rate of the article to be detected by taking the related article as a reference. Wherein the duplicate checking model is obtained by performing joint training according to training data of a plurality of mutually independent article databases, so that the articles of the duplicate checking model are more comprehensive, the duplicate checking range is expanded, the majority of researchers, college students and teachers do not need to switch different duplicate checking platforms to perform duplicate checking on the articles, the duplicate checking efficiency is improved, and in addition, the duplicate checking efficiency is improved. The related articles of the to-be-queried article can be quickly screened out through the duplicate checking model, the subsequent repetition rate detection range of the to-be-detected article is narrowed, and the article duplicate checking accuracy can be improved.

The invention discloses a data calculation and deduplication method, system, server and storage medium, which are suitable for data deduplication in big data. The method provided by the present invention includes: receiving a call request, using dubbo components to perform load balancing; parsing the request, and creating a corresponding number of redis data storage bitmaps on the server according to the preset deduplication level parameters in the request; Obtain the deduplication content parameter and the deduplication level parameter in the request, and calculate the deduplication result through the Bloom Filter algorithm. When the deduplication level is greater than 1 and the deduplication result returns a value of 0, recalculate a Group hash function, and deduplication through Bloom Filter algorithm again. In the present invention, the dubbo component is used for load balancing, and at the same time, according to the preset deduplication level, the count deduplication of the corresponding level is performed through the Bloom Filter algorithm, which can not only ensure that the data is processed efficiently and quickly, but also greatly reduce the probability of data miskilling and improve deduplication accuracy.

The invention discloses an oil well indicator diagram data duplicate checking method which comprises the following steps: acquiring indicator diagram data of all oil wells in a well area within one day, and defining effective indicator diagram data; grouping and screening the indicator diagram data of all oil wells in one day, and counting the quantity of grouped effective indicator diagram data to obtain a repeated indicator diagram data sample and a repeated indicator diagram data test sample set; screening to-be-checked duplicate indicator diagram data of all oil wells according to the duplicate indicator diagram data inspection sample set to obtain all duplicate indicator diagram data and a duplicate indicator diagram data result set; according to the repeated indicator diagram data in the repeated indicator diagram data result set and the well names thereof, obtaining a repeated indicator diagram statistical result set based on the single well names; obtaining a well name association relation table of the repeated indicator diagram data according to the repeated indicator diagram data in the repeated indicator diagram data result set and different well repetition conditions thereof; and according to the well name association relation table of the repeated indicator diagram data, obtaining all different well repeated records of each piece of repeated indicator diagram data.

The invention discloses an intelligent sales performance assessment method based on a CRM system, and the method comprises the steps: collecting a plurality of sales behavior data in a plurality of time periods, and the parameters of the behavior data comprising the unit price x of a product, the number s of products, the clue y of the product, and the clue conversion rate b of the product; inputting the plurality of behavior data into a CRM system to obtain final sales performance A in a plurality of time periods, the CRM system comprising: a normalization module for normalizing parameter values of the plurality of behavior data to obtain normalized values of the parameter values of the plurality of behavior data; and a performance calculation module used for calculating the normalized values of the parameter values of the behavior data through a specific gravity formula to obtain a final sales performance A.

The invention discloses a data duplicate checking method and a data duplicate checking device. The data duplicate checking method is applied to a system comprising a source database, a client, a cacheand a result storage database. The method comprises: at a first moment, obtaining a duplicate checking request for data to be subjected to duplicate checking, the data to be subjected to duplicate checking having a unique identifier; for the to-be-duplicated data, judging whether a corresponding unique identifier exists in the result storage database or not; when judging that the unique identifier exists, obtaining a duplicate checking moment corresponding to the unique identifier; obtaining change data in a source database between a duplicate checking moment and a first moment, and performing duplicate checking comparison with the data to be subjected to duplicate checking; and storing a duplicate checking comparison result into the result storage database. The data duplicate checking efficiency can be improved, high duplicate checking accuracy can be guaranteed, and the method and the device can be suitable for first duplicate checking and subsequent multiple duplicate checking at the same time, that is, the efficiency of multiple duplicate checking in the first duplicate checking and business process can be improved.

The invention relates to the technical field of artificial intelligence, and discloses a resume duplicate checking method and device, equipment and a medium. The method comprises: acquiring a to-be-duplicated resume; performing word segmentation according to the to-be-duplicated resume, and performing hash signature matrix calculation on a word segmentation result to obtain a to-be-duplicated hashsignature matrix; according to the to-be-duplicated hash signature matrix, and carrying out similar resume query from a resume library according to information classification to obtain a candidate resume set; respectively constructing a resume pair feature vector for the to-be-duplicated resume and each resume in the candidate resume set to obtain a plurality of to-be-predicted resume pair feature vectors; inputting the to-be-predicted resume pair feature vectors into the classification prediction model for similarity probability prediction to obtain probability prediction values of the to-be-predicted resume pair feature vectors; and determining a target repeated resume pair according to the probability prediction values of the plurality of to-be-predicted resume pair feature vectors. According to the invention, the duplicate checking efficiency is improved, similar rules do not need to be set manually, and the accuracy of determining the target repeated resume pairs is guaranteed.

The invention discloses an intelligent sales performance assessment method based on a CRM system, and the method comprises the steps: collecting a plurality of sales behavior data in a plurality of time periods, and the parameters of the behavior data comprising the unit price x of a product, the number s of products, the clue y of the product, and the clue conversion rate b of the product; inputting the plurality of behavior data into a CRM system to obtain final sales performance A in a plurality of time periods, the CRM system comprising: a normalization module for normalizing parameter values of the plurality of behavior data to obtain normalized values of the parameter values of the plurality of behavior data; and a performance calculation module used for calculating the normalized values of the parameter values of the behavior data through a specific gravity formula to obtain a final sales performance A.

The invention discloses an enterprise name duplicate checking method and device. The method comprises the following steps of: searching a second enterprise name matched with a first enterprise name tobe subjected to duplicate checking by utilizing ES; performing word segmentation on the first enterprise name and the second enterprise name according to structural elements, wherein the structural elements comprise administrative regions, company description and organization forms, and the company description comprises company word sizes and industry description; comparing each structural element in the first enterprise name with each structural element in the second enterprise name, and determining a first similarity corresponding to the administrative region, a second similarity corresponding to the company description and a third similarity corresponding to the organization form; determining the total similarity between each second enterprise name and the first enterprise name based on the first similarity, the second similarity and the third similarity; and determining the second enterprise name corresponding to the total similarity meeting the preset condition as the enterprisename which is the same as the first enterprise name. According to the invention, the duplicate checking precision and the duplicate checking efficiency can be improved.

The invention discloses a duplicate checking analysis method for project application data, which is characterized in that title classification, format merging and duplicate checking analysis are performed, the duplicate checking analysis is sequentially divided into three projects of character comparison, table comparison and file comparison, the project application data is classified, the duplicate checking efficiency is improved, the application is divided into character, table and file parts, and the duplicate checking efficiency is improved. The method comprises the following steps: respectively comparing and finally combining to generate a duplicate checking document for workers to carry out manual screening and comparison and provide reference, when duplicate checking is carried out on project application data, for different application documents of the same project, an optimal template scheme can be appointed, and when later comparison is carried out, the difference between the optimal template scheme and the optimal template scheme is independently divided into a column for comparison and extraction.

The invention relates to the technical field of digital medical treatment. The invention discloses a structured medical record duplicate checking method which comprises the following steps: acquiring a structured medical record, filtering the structured medical record to obtain medical record data, and extracting one or more keywords in the medical record data; extracting 64-bit fingerprint features of the keywords, and performing weighted accumulation on the 64-bit fingerprint features corresponding to the keywords to obtain a 64-bit feature sequence string of the structured medical record; dividing the 64-bit feature sequence string into continuous 4 sections of 16-bit sub-sequence strings, generating a query statement according to the 4 sections of 16-bit sub-sequence strings of the structured medical record, the medical record category and the disease diagnosis code, and obtaining a query result from a medical record database based on the query statement; and determining the Hamming distance between the 64-bit feature sequence string of the structured medical record and the 64-bit feature sequence string of the medical record contained in the query result, and determining whether repeated medical records are queried according to the Hamming distance. According to the method, similar structured medical records can be quickly positioned, and the duplicate checking efficiency is higher.

The invention provides a text duplicate checking method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an original text data set, building a synonym library and a word weight library based on the original text data set, and enabling the synonym library to correspond to the word weight library; on the basis of a synonym library, performing synonym replacement on feature words of each original text in the original text data set to obtain a replaced text; based on the word weight library, performing fingerprint extraction on the replaced text to obtain a fingerprint about the replaced text; based on the fingerprint of the replaced text and the original text corresponding to the replaced text, a fingerprint database about the original text is constructed, and the fingerprint of the replaced text corresponds to the original text; and determining a fingerprint of the to-be-duplicated text, and performing duplicate checking on the to-be-duplicated text based on the fingerprint database and the fingerprint of the to-be-duplicated text. According to the method and the device, the text duplicate checking accuracy and efficiency are improved.

The embodiment of the invention provides a data processing method and device, computer equipment and a computer readable storage medium, and the method comprises the steps: determining whether a cloud end has left-over data or not if the left-over data exists in a pre-sending area during data reporting, wherein the left-over data comprises data which is not deleted after the pre-sending area reports the data to the cloud end at the previous time; and if the left-over data does not exist in the cloud end, sending the to-be-reported data to the pre-sending area to instruct the pre-sending area to report the left-over data and the to-be-reported data to the cloud end. According to the embodiment of the invention, whether the left-over data exists in the cloud end or not can be determined when the left-over data exists in the pre-sending area, so that duplicate checking of the left-over data is realized, the duplicate checking efficiency can be improved, and the performance loss of computer equipment can be reduced; when the left-over data does not exist in the cloud end, the left-over data and the to-be-reported data can be reported to the cloud end together from the pre-sending area, so that the data reporting efficiency is improved.

The invention discloses a video duplicate checking method and device, a storage medium and a computer program product. The method comprises the following steps: acquiring N to-be-processed video frames and L reference video frames; the N to-be-processed video frames are determined from target videos needing duplicate checking, and the L reference video frames are determined from H reference videos used for duplicate checking comparison; based on the image feature similarity between each to-be-processed video frame and each reference video frame, determining M reference video frames from the L reference video frames as similar video frames; one similar video frame is similar to at least one video frame to be processed; determining K reference videos from the H reference videos as similar videos according to the M similar video frames; any similar video comprises at least one similar video frame; determining the audio feature similarity between the target video and each similar video, and judging whether the target video is repeated with each similar video according to the audio feature similarity; the video duplicate checking efficiency can be improved.

The embodiment of the invention discloses a method, device, equipment and storage medium for plagiarism checking of papers. The method for checking plagiarism of the paper comprises: providing at least two optional plagiarism checking platforms on the plagiarism checking display interface, and the introduction information of each optional plagiarism checking platform, so that users can choose from at least two optional plagiarism checking platforms according to their needs At least one plagiarism checking platform is used as the plagiarism checking platform to be used, the introduction information includes charging standards, and the plagiarism checking platform to be used shares the papers provided by users; the payment amount is determined according to the charging standards of the papers and each plagiarism checking platform to be used, and Complete the payment at one time, so that each plagiarism check platform to start the paper plagiarism check operation. The embodiment of the present invention provides at least two optional plagiarism checking platforms for users to choose, and the standby plagiarism checking platform shares the papers provided by the user, and completes the payment at one time when paying, so that the user does not need to use multiple plagiarism checking platforms separately Submit papers multiple times and pay multiple times, thereby saving the time for plagiarism checks and improving the efficiency of plagiarism checks.